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IMPACT  OF  THE  ARMY  CONTINUING  EDUCATION  SYSTEM  (ACES)  ON 
SOLDIER  RETENTION  AND  PERFORMANCE 

PHASE  I:  PLAN  DEVELOPMENT 

EXECUTIVE  SUMMARY 


Requirement: 

The  U.S.  Total  Army  Personnel  Command  (PERSCOM)  requested  an  evaluation  of  the 
Army  Continuing  Education  System  (ACES)  to  demonstrate  its  value  in  the  areas  of  retention 
and  soldier  performance  for  enlisted  soldiers.  The  evaluation  consists  of  two  phases.  Phase  I  is 
the  development  of  an  evaluation  plan  to  collect  and  analyze  the  data.  Phase  II  consists  of  data 
collection  and  analysis.  This  report  describes  the  planning  effort  that  was  conducted  in  Phase  I  of 
the  evaluation. 


Procedure: 

The  effort  included  a  review  of  the  relevant  research  and  evaluation  literature  and  the 
preparation  of  evaluation  and  database  development  plans.  The  literature  review  covered  relevant 
reports  in  both  the  military  and  civilian  research  literatures.  Military  research  directly  evaluated 
ACES  components,  comparable  programs  in  other  Services,  or  Department  of  Defense  (DoD)- 
wide  continuing  education  programs.  The  civilian  literature  provided  information  regarding 
analogous  programs. 

The  evaluation  plan  was  based  on  the  results  of  the  literature  review  and  on  discussions 
with  PERSCOM  regarding  the  nature  of  specific  ACES  programs.  The  plan  assesses  the  effect  of 
five  selected  ACES  programs  on  soldier  retention  and  performance.  The  planned  analysis  of 
retention  considers  reenlistment  and  early  attrition,  while  the  analysis  of  performance  focuses  on 
promotion  and  reclassification.  In  addition,  the  planned  evaluation  includes  a  cost-effectiveness 
analysis  of  ACES  programs. 

The  preparation  of  the  database  development  plan  began  with  a  thorough  assessment  of 
the  data  required  for  the  evaluation.  This  effort  identified  specific  criteria  by  which  to  assess  and 
select  data  sources  and  evaluated  relevant  data  sources  according  to  these  criteria.  The  selected 
data  sources  were  judged  to  provide  the  richest,  most  efficient,  and  most  relevant  information  for 
use  in  the  evaluation  study.  The  final  step  developed  detailed  guidelines  for  data  procurement 
and  for  the  creation  of  an  analytic  database  to  support  the  evaluation. 


Findings: 

The  research  literature  provides  limited  coverage  of  continuing  education  programs. 
Results  indicate  that  those  who  participate  in  continuing  education  tend  to  be  better  qualified  and 
more  highly  motivated  soldiers  (or  employees)  than  those  who  don’t.  A  positive  effect  of 
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participation  on  performance  is  reflected  in  personal  opinions,  promotion  rates,  and  actual 
performance  ratings.  Regarding  retention,  the  research  suggests  that  participation  in  continuing 
education  increases  the  likelihood  of  reenlistment.  The  effect  remains  at  a  reduced  magnitude 
when  other  factors  are  controlled  statistically.  The  generality  of  the  findings  of  past  research  is 
limited  by  the  relatively  small  number  of  evaluation  studies,  the  focus  of  these  studies  on  a  small 
number  of  continuing  education  programs  (primarily  tuition  assistance),  the  inconsistent 
correction  for  selection  bias,  and  the  lack  of  an  overall  conceptual  model  to  guide  the  selection  of 
predictor  variables  and  interpretation  of  results. 

The  evaluation  plan  addresses  five  ACES  programs:  (a)  the  Army  Tuition  Assistance 
(TA)  Program,  (b)  the  Servicemembers  Opportunity  Colleges  Army  Degree  (SOCAD)  Program 
(c)  the  Functional  Academic  Skills  Training  (FAST)  program,  (d)  Military  Occupational 
Specialty  (MOS)  Improvement  Courses,  and  (e)  Non-commissioned  Officer  (NCO)  Leader 
Development  Courses.  The  methodological  approach  addresses  several  potential  evaluation 
problems,  including  non-random  assignment,  censored  data,  missing  data,  measurement  error, 
and  unobserved  heterogeneity.  The  database  development  plan  specifies  variables  from 
administrative  databases  and  surveys  that  measure  participation  in  ACES  programs,  assess 
retention  and  performance  outcomes,  and  identify  other  characteristics  that  should  be  used  as 
controls.  The  plan  also  identifies  data-building  procedures  that  will  result  in  an  analytically 
relevant  evaluation  database. 


Use  of  Findings: 

The  products  of  this  Phase  I  effort  -  the  literature  review,  evaluation  plan,  and  data  development 
plan  -  provide  sound  guidelines  and  procedures  for  the  Phase  II  ACES  evaluation  study. 
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INTRODUCTION 


The  Army’s  role  as  educator  began  in  1778  during  the  Revolutionary  war  when  General 
Washington  ordered  chaplains  to  teach  convalescent  soldiers  how  to  read.  However,  it  was  not 
until  the  20th  century  that  educational  benefits  became  more  widely  available  to  both  veterans 
and  active  duty  servicemembers.  The  Rehabilitation  Act  of  1919  provided  educational  assistance 
to  veterans  who  were  disabled  in  World  War  L  By  the  end  of  World  War  H,  educational  benefits 
were  offered  to  all  veterans  by  the  Servicemen’s  Readjustment  Act  of  1944,  better  known  as  the 
G.I.  Bill  of  Rights.  Further  expansion  of  benefits  for  Vietnam  Era  veterans,  passed  by  Congress 
in  1966,  offered  educational  assistance  to  active  duty  servicemembers. 

With  the  advent  of  the  All  Volunteer  Force  in  1973,  the  Military  Services  and 
Department  of  Defense  (DoD)  increased  educational  benefits  to  serve  as  an  incentive  for 
recruitment  and  to  encourage  recruits  to  select  critical  Military  Occupational  Specialties  (MOS). 
The  current  Montgomery  G.I.  Bill  (MGIB),  enacted  in  1985,  provides  up  to  36  months  of 
educational  assistance  that  can  be  used  by  both  veterans  and  active  duty  servicemembers. 

Realizing  the  need  for  continuing  education  (CE)  among  its  members,  the  Services  and 
DoD  established  programs  to  support  the  volunteer,  off-duty  educational  activities  of  enlisted 
personnel  and  officers.  The  Army  Continuing  Education  System  (ACES)  represents  a  series  of 
programs  that  are  available  to  soldiers  and,  in  some  cases,  their  dependents  and  Army  civilian 
personnel.  The  mission  of  ACES  is  to  vigorously  promote  lifelong  learning  opportunities  to 
sharpen  the  competitive  edge  of  the  Army  by  providing  and  managing  quality  educational 
programs  and  services.  ACES  includes  the  following  programs  to  meet  the  educational  needs  of 
soldiers  and  to  help  soldiers  to  apply  the  skills  learned  in  the  Army  to  obtain  academic 
credentials  needed  for  their  later  civilian  life. 

•  The  American  Council  of  Education  Military  Evaluations  Program  reviews  formal 
military  training  courses  to  determine  the  extent  to  which  they  are  equivalent  to  college 
courses.  Soldiers  can  gain  college  credit  for  their  military  experience  through  this 
program. 

•  The  Army  Personnel  Testing  Program  provides  the  soldier  with  the  opportunity  to  take 
standardized  tests  that  are  used  for  selection  and  classification  purposes. 

•  The  Army  Tuition  Assistance  (TA)  Program  helps  soldiers  to  finance  voluntary 
participation  in  off-duty  postsecondary  educational  programs. 

•  The  Functional  Academic  Skills  Training  (FAST)  Program  provides  soldiers  with 
instmction  to  enhance  basic  skills  necessary  for  job  proficiency  or  career  progression. 

•  The  English  as  a  Second  Language  (ESL)  program  provides  education  to  increase 
language  proficiency  among  non-native  speakers. 

•  Leader  Development  Programs  provide  opportunities  to  obtain  the  skills  required  by  non¬ 
commissioned  officers  (NCO)  to  effectively  lead  their  troops. 
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•  The  High  School  Completion  Program  provides  soldiers  and  their  adult  family  members 
an  opportunity  to  earn  a  high  school  credential. 

•  The  Servieemembers  Opportunity  Colleges  Army  Degree  (SOCAD)  Program  allows 
soldiers  to  earn  a  job-related  college  degree  at  locations  on  or  near  Army  installations. 

•  The  Army/American  Council  on  Education  Registry  Transcript  System  provides  a 
transcript  that  translates  a  soldier’s  military  experience  into  civilian  terms. 

The  ACES  Program  is  administered  by  the  Education  Division,  U.S.  Total  Army  Personnel 
Command  (PERSCOM).  Education  counselors  located  at  installation  Army  Education  Centers 
coordinate  soldier  participation  in  ACES  programs. 

PERSCOM  requested  an  evaluation  of  ACES  to  demonstrate  its  value  to  the  Total  Army 
in  the  areas  of  retention  and  performance  of  enlisted  soldiers.  The  evaluation  consists  of  two 
phases.  Phase  I  is  the  development  of  evaluation  and  database  development  plans  to  collect  and 
analyze  the  data.  Phase  II  consists  of  data  collection  and  analysis.  This  report  describes  Phase  I, 
which  consists  of  three  components — a  literature  review,  an  evaluation  plan,  and  a  database 
development  plan. 

The  primary  purpose  of  the  literature  review  is  to  provide  input  for  the  evaulation  and 
database  development  plans  that  follow  it  in  this  report.  More  specifically,  the  review  of  the 
military  and  civilian  literature  on  the  impact  of  education  program  participation  on  job 
performance  and  turnover  serves  two  major  purposes. 

First,  the  review  provides  information  to  develop  a  conceptual  framework  and  a  model  to 
understand  how  participation  in  an  education  program  (like  ACES)  could  affect  job  performance 
and  turnover.  Although  the  nature  of  ACES  differs  in  many  ways  from  some  of  the  education 
programs  analyzed  in  the  literature,  both  the  goals  of  previous  studies  and  the  methods  used  to 
meet  them  are  relevant  to  this  study.  The  conceptual  model  that  builds  on  the  extant  literature 
will  help  determine  what  information  should  be  collected  for  the  evaluation.  This,  in  turn,  affects 
both  the  database  development  plan  and  the  evaluation  plan.  Also,  a  review  of  the  theory  on  the 
purpose  of  CE  programs  and  the  value  added  may  provide  insights  for  critiquing  ACES  and 
suggesting  areas  for  improvement  in  its  design. 

Second,  our  review  of  the  empirical  literature  provides  useful  information  on  many  of  the 
data  limitations  and  statistical  issues  that  previous  studies  have  encountered.  In  the  review,  we 
both  describe  and  critique  the  methods  and  statistical  techniques  used  by  various  authors  to 
address  these  issues.  This  information,  in  turn,  helps  guide  the  evaluation  and  database 
development  plans. 

The  evaluation  and  database  development  plans  build  upon  the  information  identified  in 
the  literature  review.  These  two  plans  were  developed  in  tandem  using  an  iterative  process.  On 
one  hand,  the  evaluation  plan  establishes  the  requirements  for  data  that  must  be  included  in  the 
evaluation  database.  A  major  effort  in  the  development  of  the  database  development  plan  Is 
identifying  and  evaluating  sources  of  the  data  required  by  the  evaluation  plan.  On  the  other  hand, 
the  evaluation  plan  must  take  into  account  the  constraints  brought  about  by  the  availability  and 
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quality  of  relevant  data.  If  the  information  necessary  to  evaluate  a  particular  program  is  not 
recorded,  or  is  not  recorded  accurately,  then  the  evaluation  plan  must  be  adjusted  to  reflect  this 
constraint. 

The  evaluation  plan  focuses  on  required  independent,  dependent,  and  control  variables, 
and  provides  a  very  general  description  of  potential  sources  for  these  variables.  The  database 
development  plan  that  follows  the  evaluation  plan  gives  a  detailed  description  of  both  data 
sources  and  procedures  used  to  obtain  the  required  information. 
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REVIEW  OF  RELEVANT  RESEARCH  AND  EVALUATION  LITERATURE 


The  most  relevant  literature  included  in  this  review  consists  of  research  and  studies 
conducted  by  or  for  military  agencies.  This  body  of  research  directly  evaluated  ACES 
components,  comparable  programs  in  other  Services,  or  DoD-wide  continuing  education 
programs.  In  searching  for  and  identifying  relevant  reports,  we  focused  on  studies  that 
considered  the  organizational  outcome  measures  of  interest — retention,  performance,  promotion, 
attrition,  or  recruiting.  In  addition,  we  reviewed  studies  addressing  the  reasons  people  participate 
in  CE,  and  the  individual  benefits  they  obtain  from  it.  The  majority  of  the  research  that  was 
identified  addressed  tuition  assistance  programs.  The  focus  of  previous  research  on  this  program 
may  reflect  its  relatively  high  cost  (compared  to  other  CE  programs),  its  importance  in  the 
military  education  system,  and  the  ease  with  which  participation  can  be  quantified.  In  addition, 
we  were  given  reports  describing  evaluation  research  addressing  the  Army  Basic  Skills 
Education  Program  (BSEP,  a  forerunner  of  FAST)  conducted  by  the  U.S.  Army  Research 
Institute  (ARI).  We  did  not  identify  any  research  that  evaluated  the  effects  of  educational  or 
vocational  testing  or  counseling  programs  on  recruiting,  performance,  or  retention. 

Many  of  the  studies  we  reviewed  focused  on  voluntary  education  programs  for  officers. 
Although  the  primary  focus  of  this  study  is  enlisted  soldiers,  a  review  of  the  literature  focusing 
on  officers  serves  two  purposes.  First,  the  methodology  used  in  these  studies  can  be  applied  to 
study  the  impact  of  CE  on  enlisted  personnel  behavior  and  outcomes.  Second,  the  review  of  the 
officer  literature  may  indicate  a  need  to  study  the  effect  of  ACES  participation  among  the 
officers  in  greater  detail.  This  review  of  the  research  addressing  officers  revealed  that  motivation 
for  participation  in  a  CE  program  differs  between  officers  and  enlisted  personnel. 

The  civilian  literature  also  provided  relevant  information  for  this  review.  These  reports 
were  identified  from  searches  of  bibliographic  databases,  as  well  as  the  reference  lists  from 
already  identified  reports.  Our  focus  on  specific  outcome  measures  limited  the  number  of  reports 
that  were  reviewed  in  greater  detail.  Interestingly,  the  private-sector  educational  programs  that 
were  reviewed  most  thoroughly  also  focused  on  tuition  assistance  for  postsecondary  education. 

Military  and  civilian  employers  have  similar  concerns  regarding  the  demands  of 
recruitment  and  retention  of  employees.  In  the  past,  providing  education/training  for  employees 
was  for  the  company’s  own  purpose,  to  keep  workers  up-to-date  and  productive.  This  is  no 
longer  the  case  in  today’s  job  market.  A  1998  Saratoga  Institute  study  shows  that  employee 
motivation  and  loyalty  are  tied,  in  part,  to  training  and  educational  benefits — that  individuals 
seek  their  own  development  process  and  not  one  driven  by  their  employer  (Olesen,  1999).  A 
1999  survey  by  the  Society  for  Human  Resource  Management  indicates  that  American  business 
is  responding  to  this  need.  The  survey  indicates  that  to  enhance  recruiting,  94%  of  U.S. 
companies  with  5000  or  more  employees  provide  professional  development  and  84%  offer 
educational  benefits  (Olesen,  1999).  A  large-scale  survey  of  financial  and  tuition  assistance 
benefits  among  the  Fortune  1000  companies  was  published  in  1986  (O’Neill,  1986),  showing 
that  corporate  and  military  practices  are  similar  in  terms  of  tuition  reimbursement  rates. 
Furthermore,  Turner  (1995)  found  that  since  the  O’Neill  report  was  published,  there  has  been  a 
trend  away  from  limits  being  applied  to  the  type  of  courses  for  which  employees  would  be 
reimbursed.  In  the  past,  employees  would  be  expected  to  take  courses  related  to  their  job  or 
industry.  Turner  found  that  nearly  two-thirds  of  those  employers  surveyed  reimbursed  personnel 
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for  non-job-related  courses,  particularly  if  they  were  part  of  a  degree  program.  These  trends  in 
the  private  sector  can  act  as  a  benchmark  to  the  military,  since,  theoretically,  these  companies 
and  the  military  are  in  competition  to  recruit  the  same  people. 

However,  differences  in  the  nature  of  military  employment  and  private  sector 
employment  limit  the  ability  to  generalize  findings  from  the  civilian  literature  to  the  military 
There  are  three  major  differences  between  the  two  sectors  that  are  relevant  to  this  study. 

•  First,  the  nature  of  the  contractual  relationship  between  employer  and  employee  is 
different.  In  the  private  sector,  the  employment  relationship  can  be  terminated  at  will.  In 
the  military,  the  soldier  and  the  military  enter  a  contractual  relationship  that  the  soldier 
remain  in  the  military  through  the  end  of  the  enlistment  term.  Consequently,  the 
relationship  between  CE  participation  and  turnover  in  the  military  and  in  the  private 
sector  may  be  different. 

•  Second,  the  personnel  system  in  the  military  is  a  closed  hierarchy.  That  is,  leadership 
positions  are  filled  from  lower  rahks.  In  the  private  sector,  management  positions  may  be 
filled  from  promotions  within  the  firm,  or  may  be  external  hires.  Consequently, 
promotions  as  a  measure  of  job  performance  are  more  appropriate  for  the  military  than 
for  the  civilian  sector.  Likewise,  measures  of  performance  in  the  civilian  sector  (e.g., 
salaries)  may  not  be  excellent  measures  of  performance  in  the  military,  where  there  is 
less  variation  in  pay,  and  pay  is  not  directly  tied  to  performance. 

•  Third,  the  military  offers  excellent  educational  benefits  for  those  who  leave  the  military 
while  private  sector  firms  generally  do  not.  For  example,  the  military  aggressively 
advertises  the  G.I.  Bill  and  Army  College  Fund  (ACF)  benefits,  which  the  soldier  use 
primarily  after  leaving  the  military. 

Despite  these  differences  in  the  characteristics  of  the  employment  relationship  and  personnel 
systems,  a  review  of  the  civilian  literature  provides  useful  information  on  the  methods  for 
evaluating  the  value  added  by  CE  programs. 

The  remainder  of  this  review  summarizes  previous  empirical  studies  designed  to  evaluate 
the  effects  of  participation  in  military  and  civilian  continuing  education  programs.  In  the  next 
section  of  the  report,  we  review  the  literature  examining  the  motivations  for  participation  in 
continuing  education  programs  and  the  characteristics  of  people  who  participate.  The  following 
section  reviews  the  literature  examining  the  benefits  of  these  programs  for  the  organization, 
focusing  on  recruiting,  performance,  and  turnover.  The  report  continues  with  a  discussion  of 
empirical  and  methodological  problems  and  issues  encountered  in  the  evaluations  of  these 
programs.  The  final  section  summarizes  the  results  of  the  review  and  discusses  the  implications 
on  the  design  of  an  evaluation  of  ACES. 

Factors  Predicting  Participation  in  Continuing  Education  Programs 

This  section’s  focus  is  on  the  factors  that  contribute  to  an  individual’s  propensity  to 
participate  in  voluntary,  off-duty  educational  opportunities.  It  is  important  to  know  who 
participates  in  CE  programs  and  why.  First,  the  fact  that  participants  are  a  self-selected  sample 
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may  bias  the  data  used  to  establish  a  relationship  between  CE  participation  and  measures  of 
retention  or  performance.  Understanding  more  about  which  individuals  participate  and  why  will 
ameliorate  some  of  the  problems  associated  with  this  bias.  Second,  it  is  useful  to  know 
characteristics  about  the  individuals  that  participate.  Knowledge  of  these  characteristics  drives 
the  research  questions  and  provides  expectations  that  can  validate  or  invalidate  models. 

This  section  begins  by  reviewing  the  literature  analyzing  the  individual  motivations  to 
participate  in  CE  programs.  Because  there  generally  has  been  no  well-defined  theory  driving  the 
studies  reviewed  below,  the  models  of  participation  have  been  defined  more  by  the  data.  Because 
they  lack  theory-driven  designs,  these  studies  tend  to  use  typical  demographic  characteristics  to 
help  predict  participation. 

Motivations  Associated  with  CE  Participation 

It  is  important  to  understand  that  motivation  varies  between  officers  and  enlisted 
personnel,  and  likewise  between  military  and  civilian  participants.  A  review  of  the  civilian 
literature  is  included  as  a  point  of  comparison.  Though  the  civilian  literature  on  the  value  of 
participation  in  voluntary  education,  particularly  in  the  business  sector,  is  generally  more 
hortatory  than  empirical,  some  empirical  research  exists.  This  literature  examines  the  reasons 
professionals  do  or  do  not  participate  in  continuing  and  voluntary  education. 

Military  studies.  What  motivates  a  soldier  to  participate  in  off-duty,  voluntary  education? 
Soldiers  participate  to:  (a)  improve  promotion  potential,  (b)  increase  “social  standing”  or 
personal  satisfaction,  (c)  increase  their  ability  to  change  careers  or  MOS,  or  (d)  improve  earnings 
potential  in  the  private  sector.  Understanding  a  soldier’s  motivation  for  participating  can  help 
focus  the  research  design.  For  example,  if  one  believes  that  participation  in  CE  is  driven  mainly 
by  one’s  desire  to  improve  promotion  potential,  and  thus  military  earnings  potential,  then  one 
would  expect  that  CE  participation  would  lead  to  higher  retention.  On  the  other  hand,  if  one 
believes  that  CE  participation  is  to  improve  earnings  potential  in  the  private  sector,  then  one 
would  expect  CE  participation  to  lead  to  lower  rates  of  retention.  So,  understanding  motivation 
can  result  in  more  properly  specified  retention  models. 

Brauchle  (1998)  showed  that  an  individual’s  motivation  to  participate  in  off-duty 
education  is  dependent  on  a  number  of  factors.  First  of  all,  the  military  culture  itself  values 
education  and  is  encouraging  to  servicemembers  who  use  the  benefits  allotted  to  them.  However, 
a  person’s  ability  to  use  those  benefits  is  greatly  determined  by  opportunity  -  opportunity  to 
participate  is  not  constant  throughout  one’s  military  career  but  varies  based  on  location,  job,  and 
military  specialization.  Brauchle  also  found  that  individual  motivation  varies  throughout  one’s 
military  career.  He  notes  that  servicemembers  receive  considerable  external  motivation  to 
participate  in  off-duty  education  early  in  their  career.  As  they  progress,  that  motivation  becomes 
more  internalized. 

The  only  other  study  included  in  this  review  that  focuses  on  servicemember’s  motivations 
for  participating  in  CE  used  the  Participation  Reasons  Scale  (PRS)  to  examine  the  reasons  Army 
Engineers  (officers)  participated  in  off-duty,  civilian  education.  The  PRS,  was  created  and  tested 
in  the  late  1970s  to  help  develop  education  participation  models.  It  was  originally  validated  using 
businessmen  and  women  as  subjects  (Catlin,  1982).  Grzyb’s  (1997)  analysis  identified  five 


7 


reasons  that  Army  engineers  (ranked  lieutenant,  captain  or  major)  participated  in  CE: 

(a)  professional  improvement/development,  (b)  personal  development  and  job  security, 

(c)  improvement  of  service  to  customers,  (d)  professional  identity/perspective,  and 
(e)  competence  and  collegial  interaction.  Generally,  military  engineers  resembled  other 
professions  (judges,  physicians,  etc.)  using  this  scale.  However  there  were  some  differences.  For 
example,  variables  measuring  leadership  and  functional  roles,  educational  level  and  preparation, 
occupational  specialty,  rank  and  years  performing  duties,  contrary  to  the  expectation,  were  not 
associated  with  an  Army  Engineer’s  reasons  for  participating  in  continuing  education.  Grzyb, 
using  structured  interviews  to  complete  his  research,  concluded  that  Army  Engineers  shared 
cultural  elements,  even  set  apart  from  the  Army  as  a  whole,  that  influenced  their  attitudes  and 
motivation  toward  participating  in  voluntary  education.  For  example,  leaders  repeatedly 
emphasized  participation  verbally,  in  writing  and  by  modeling  behavior  by  participating  in  off- 
duty  education  themselves.  The  research  suggests  that  organizational  culture  (shared  values) 
creates  norms  that  can  contribute  to  an  individual’s  propensity  to  participate  in  voluntary 
education. 

Civilian  studies.  The  civilian  literature  on  adult  voluntary  education  suggests  and  often 
assumes  that  participation  is  internally  motivated.  Recent  research  suggests  that  a  person’s 
motivation  to  participate  in  voluntary  education  is  complex  and  multi-dimensional,  often 
controlled  by  external  forces.  For  instance.  Stalker  (1993)  found  that  employees  given 
educational  opportunities  viewed  the  benefit  as  a  favor  bestowed  on  them  by  authority,  and  that 
the  opportunity  to  use  these  benefits  was  viewed  by  both  the  institution  and  the  employee  as  a 
privilege  (participants  are  passive  recipients).  As  Stalker  notes,  some  subtle  organizational 
factors  may  mandate  participation  even  when  participation  is,  on  the  surface,  considered 
voluntary.  This  is  relevant  to  the  military  since  the  research  analyzing  participation  in  these 
programs  generally  assumes  internal  motivation  in  order  to  attain  personal  goals  such  as 
promotion,  or  simply  self-enhancement.  What  is  revealed  is  that  reasons  for  participation  are 
more  difficult  to  model. 

Darkenwald  and  Valentine  (1985)  report  that  earlier  studies  developing  theories  about 
voluntary  education  participation  focused  on  identification  of  motivations  and  typologies  of 
learning  behavior  (Boshier,  1971;  Burgess,  1971;  and  Boshier  &  Collins,  1985).  Martindale  and 
Drake  (1989)  note,  however,  that  these  studies  failed  to  develop  theories  that  could  help 
practitioners  predict  participation.  Earlier  studies  also  failed  to  analyze  the  deterrents  to 
participation  in  voluntary  education  opportunities,  despite  the  fact  that  many  studies  concluded 
that  a  deterrent  constmct  is  fundamental  to  models  of  participation  (Martindale  &  Drake,  1989; 
Cross,  1981;  Darkenwald  &  Merriam,  1982).  Darkenwald  and  Valentine  (1985)  and  Scanlan  and 
Darkenwald  (1984)  developed  survey  tools  incorporating  deterrent  constructs.  The  Deterrents  to 
Participation  Scale  (DPS)  and  the  generic  form  of  this  instrument  (DPS-G)  included  factors,  such 
as  lack  of  confidence,  low  personal  priority  and  time  constraints  as  reasons  that  individuals 
refrain  from  participating  in  off-duty  educational  opportunities.  These  instruments  have  been 
used  in  both  civilian  as  well  as  military  populations  to  measure  deterrents  to  voluntary  education 
participation. 

Focusing  on  the  deterrent  effect  does  not  explain  what  the  factors  are  that  contribute  to  a 
person’s  participation  in  voluntary  education.  There  are  numerous  studies  analyzing  the  reasons 
for  and  motivations  of  professionals  to  participate  in  life-long  learning.  An  instrument  that  has 
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been  used  consistently  in  empirical  studies  to  ascertain  motivation  is  the  PRS,  noted  previously. 
Cervero  (1981),  Groteleuschen  (1985)  and  Moore,  Bennet,  Knox,  and  Kristofco  (1994)  used  the 
PRS  to  look  at  factors  contributing  to  continuing  education  among  physicians,  for  example.  Most 
people  assume  that  medical  professionals  are  involved  in  continuing  education  in  order  to  keep 
up  with  new  developments  in  medical  techniques,  research  and  technology,  and  to  provide  better 
medical  care.  Cervero  found  in  analyzing  physicians  responses  on  the  PRS  that  the  reasons  were 
more  complex,  showing  that  physicians  participate  to  enhance  their  personal  and  professional 
position,  to  interact  with  colleagues  more  and  to  understand  themselves  in  their  profession.  There 
have  been  other  studies  using  participation  scales  looking  at  other  health  professionals,  such  as 
nurses  (DeSilets,  1995)  and  veterinarians  (Hamish,  1980).  The  results  are  similar 

Catlin  (1982),  using  the  same  methodological  approach  and  PRS  instrument,  analyzed 
why  Michigan  judges  participate  in  continuing  education,  but  included  a  correlation  analysis  to 
determine  which  personal  and  professional  characteristics  correlated  with  the  participation 
factors.  Catlin  found  that  there  were  three  factors  that  emerged  from  the  analysis: 

(a)  professional  perspective,  (b)  competence,  and  (c)  collegial  interaction.  This  is  consistent  with 
the  participation  rationale  for  other  professionals.  The  correlation  analysis  showed  that  women 
appear  to  participate  in  voluntary  education  more  than  men,  to  maintain  an  acceptable  level  of 
competence  and  judicial  skill.  The  analysis  also  revealed  that  newer  judges  place  a  greater 
importance  on  voluntary  education  for  the  same  reason,  compared  to  those  with  more  time  on  the 
bench.  Though  there  are  obvious  differences  among  the  professions  of  judge,  physician  and 
soldier,  a  finding  like  this  could  nevertheless  be  applicable  to  the  military  setting.  It  may  suggest 
that  newer  enlistees  will  more  likely  use  their  educational  benefits  in  order  to  maintain 
competence  and  quality  and  that  higher  ranked  enlistees  may  participate  in  voluntary  education 
for  other  reasons. 

Other  Characteristics  Associated  with  CE  Participation 

These  studies,  above,  show  that  motivation,  whether  internalized  or  externalized,  can 
explain  a  servicemember’s  reasons  for  participating  in  continuing  education.  Other  research  that 
attempts  to  understand  or  predict  CE  participation  is,  at  least  on  the  surface,  less  theory  driven. 

In  lieu  of  theoretical  explanations  to  accurately  specify  models,  analyses  in  the  following  studies 
tend  to  use  standard  demographic  and  personal  characteristics  to  explain  CE  participation,  with 
very  little  explanation  of  why  those  characteristics  are  included  in  the  model.  For  example, 
Becerra  (1983)  suggested  that  women  and  minorities,  have  a  greater  tendency  than  white  men  to 
view  the  military  as  a  vehicle  for  upward  socioeconomic  movement.  As  such,  it  makes  sense  to 
include  race  and  sex  in  models  explaining  CE  participation.  As  Boesel  and  Johnson  (1988)  note, 
“one  would  expect  to  see  a  tendency  among  women  and  minority  members  to  take  advantage  of 
the  educational  opportunities  afforded  by  TA  [tuition  assistance]  as  a  means  of  upward  mobility” 
(p.  11).  Additionally,  many  of  the  studies  that  analyze  the  effects  of  participation  in  educational 
benefits  on  retention  and  performance  look  also  at  what  factors  predict  participation  in  the  first 
place.  This  is  done,  primarily,  to  control  for  selection  bias.  The  result  is  that  these  studies  lack 
clarification  of  why  certain  variables  are  included  to  explain  CE  participation.  Clearly,  more 
theory-driven  research  is  needed  to  explain  CE  participation.  Nevertheless,  available  research 
identifies  certain  characteristics  that  can  help  predict  CE  participation. 


9 


Several  studies  have  shown  that  military  service  itself  has  had  a  positive  effect  on 
educational  attainment  for  veterans  (Binkin,  Eitelberg,  Schexnider,  &  Smith,  1982;  Kolstad, 
1986,  Mason,  1970).  Cohen,  Segal,  and  Jemme  (1986),  found  that  the  higher  the  rank  one 
achieved  in  the  military,  the  higher  the  level  of  education  that  was  eventually  achieved.  But  these 
results  seem  to  be  confounded  by  the  fact  that  promotions  are  partly  determined  by  educational 
level  achieved.  Others  have  found  that  when  comparing  educational  attainment  level  of  white 
servicemembers  to  their  civilian  counterparts,  those  in  the  military  attained  less  education,1  but 
this  did  not  hold  true  for  black  and  Hispanic  servicemembers,  who  average  much  more  education 
than  their  civilian  contemporaries  (Fredland  &  Little,  1984).  Fredland  and  Little  also  found  that 
white,  black  and  Hispanic  servicemembers  had  higher  educational  aspirations  than  their  civilian 
contemporaries.  In  terms  of  motivation  to  participate  in  educational  opportunities,  many  argue 
that  people  are  attracted  to  the  military  primarily  for  the  purpose  of  getting  an  education.  Some 
servicemembers  view  their  military  service  as  one  and  the  same  with  their  educational 
aspirations  (Kolstad,  1986).  While  this  view  may  be  widespread  among  servicemembers, 
researchers  have  also  found  that  there  are  important  intervening  variables  that  influence 
educational  attainment  among  servicemembers. 

For  example,  Wright  (1989)  found  that  the  mother’s  education,  the  father’s  occupation, 
high  school  grade  point  average,  student  aptitude,  student  high  school  program,  and  the 
individual’s  reason  for  entering  the  military  were  all  individually  significant  predictors  of  a 
servicemember  s  educational  attainment.  When  these  factors  are  considered  together,  the  two 
best  predictors  of  the  level  of  educational  attainment  of  military  enlistees  were  the  mother’s 
education  and  her  educational  aspirations  for  the  enlistee.  When  comparing  servicemembers  with 
their  civilian  contemporaries,  Wright  found  that  there  was  very  little  difference  in  the  factors  that 
influenced  educational  attainment.  The  author  suggested  that  the  military  should  take  these 
characteristics  into  account  when  utilizing  educational  incentives  for  enlistment  or  retention. 

Brauchle  (1998)  derived  interesting  results  by  analyzing  both  short-  and  long-term 
participation  in  educational  benefits  as  dependent  variables.2  For  both  measures  of  participation, 
women  were  more  likely  than  men  to  participate,  at  a  rate  of  1.5:1.  Single  servicemembers  were' 
more  likely  to  participate  in  the  short  term  than  married,  but  in  the  long  term,  married  members 
were  more  likely  to  participate.  Army  and  Air  Force  servicemembers  are  more  likely  to 
participate  than  sailors  and  Marines  (this  is  probably  due  to  shipboard  deployments).  However, 
Army  members  participated  in  short-term  education  at  higher  rates  than  Air  Force  members,  and 
Air  Force  members  were  much  more  likely  to  participate  in  long-term  education  than  Army 
enlistees.  The  results  of  this  study  should  be  interpreted  carefully,  because  many  of  the 


Fredland  and  Little  (1984)  note  that  “comparing  educational  levels  of  young  servicemen  with  those  of  civilians  of 
similar  ages  is  biased  if  the  data  are  truncated  by  age.  If  men  under  22  are  examined,  members  of  the  military  clearly 
cannot  have  completed  as  much  education  as  civilians  who  went  directly  from  high  school  to  college,  and  even  to 
graduate  school”  (p.  212). 

2  Short-term  participation  assessed  whether  or  not  the  person  had  attended  a  civilian  college  during  the  previous 
year.  The  long-term  voluntary  education  variable  was  created  based  on  survey  responses  to  questions  concerning 
education  level  at  time  of  entry  (into  the  military)  and  the  education  level  possessed  at  the  time  the  survey  was 
completed.  Long-term  participation  was  defined  as  an  increase  in  education  level  (from  entry  to  the  time  of  the 
survey),  with  the  restriction  that  the  individual  had  completed  at  least  “some  college.” 
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variables — including  long-term  participation,  reenlistment  intentions,  and  marital  status — are 
related  to  time  in  service. 

Two  other  studies  that  analyze  the  characteristics  associated  with  CE  participation  focus 
on  Navy  and  Marine  Corps  officers.  In  terms  of  motivation  for  CE  participation  there  is  some 
convergence  between  officers  and  enlisted  servicemembers  (i.e.,  promotion  potential,  individual 
aspiration,  etc.),  however,  there  are  important  differences.  In  general,  studies  show  that  enlisted 
personnel  are  more  likely  to  participate  in  TA  programs.  For  example,  Boesel  and  Johnson 
(1988)  found  that  Army  enlisted  personnel  are  more  than  three  times  as  likely  to  participate.  This 
is  most  likely  because  officers  incur  further  obligation  to  the  military  if  they  use  CE  benefits, 
whereas  enlisted  personnel  do  not.  Additionally,  officers  tend  to  have  college  degrees  already. 
Boesel  and  Johnson  indicated  that  personnel  with  college  degrees  (whether  enlisted  or  from  the 
officer  corps)  participate  in  TA  programs  at  much  lower  rates. 

Fuchs  (1996),  in  trying  to  predict  which  Naval  officers  choose  fully-funded  graduate 
education,  found  that  those  with  better  undergraduate  records  and  with  a  more  technical 
background  were  more  likely  to  seek  and  be  selected  for  graduate  education.  Fuchs  found  that 
married  officers  tended  to  pursue  graduate  education  at  higher  rates.  Additionally,  those  officers 
who  were  recommended  to  receive  a  promotion  earlier  than  average  (as  an  Ol  or  an  02),  were 
more  prone  to  seek  graduate  education.  Wielsma  (1996)  conducted  a  very  similar  study  focused 
on  Marine  Corps  officers.  He  found  that  better  performers  and  women  were  more  likely  to 
participate  in  graduate  education  than  lower  performers  and  men.  He  also  found  that 
commissioning  source  was  a  good  predictor  of  graduate  school  participation.  Naval  Academy 
graduates  were  more  likely  to  participate  than  those  commissioned  any  other  way. 

Conclusion 

One  of  the  main  findings  here  is  that  more  theory  development  is  needed  to  understand 
who  participates  in  CE  and  why.  Lack  of  theoretical  understanding  may  not  only  lead  to  poorly 
specified  models,  but  also  to  wrong  conclusions.  For  example,  Garcia,  Joy,  and  Reese  (1998)  in 
their  study  of  the  Navy's  Voluntary  Education  program,  found  that  education  program 
completion  rates  are  lower  for  junior  sailors  compared  to  senior  sailors.  Consequently,  the 
authors  recommended  that  the  Navy  limit  enrollment  of  junior  sailors.  Theory  would  suggest, 
though,  that  the  program's  value  added  to  the  Navy  may  be  greater  for  junior  enlisted  (e.g.,  Els 
and  E2s)  than  for  more  senior  enlisted  (e.g.,  E3s  and  E4s).  For  example,  consider  two  sailors  in 
their  first  enlistment — an  early  career  El  and  an  E4  near  the  end  of  his  or  her  enlistment.  Given 
that  both  sailors  have  the  same  probability  of  reenlisting,  the  Navy  would  have  a  longer  time 
period  over  which  to  recoup  investment  in  the  soldier  who  participates  earlier  in  the  program. 
Thus,  while  Garcia  et  al.  found  that  limiting  enrollment  for  junior  soldiers  may  be  the  Navy's 
recommended  course,  theory  suggests  that  the  opposite  may  be  true. 

Table  1  depicts  the  characteristics  and  motivations  of  those  who  participate  in  CE 
programs,  based  on  the  literature  reviewed  in  this  section. 
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Table  1 

Who  Participates  in  Continuing  Education  and  Why 


Characteristics  Predicting 
CE  participation 

Motivations  to 
participate  in  CE 

From  military  studies 

•  Military  service 

•  Achieving  higher  rank 

•  Race/ethnicity 

•  Mother’s  education 

•  Mother’s  educational 
aspirations  for  enlistee 

•  High  School  GPA 

•  Student  aptitude 

•  High  School  program 

•  Individual’s  reason  for 
entering  military  service 

•  Sex 

•  Marital  status 

•  Military  branch 

•  Level  of  contentment 
with  military  life 

•  Promotion  status 

•  Military  performance 

•  Source  of  commission 

•  Military  culture/norms 

•  Opportunity  to 
participate 

•  Professional 
improvement/developm 
ent 

•  Personal  development 

•  Job  security 

•  Improvement  of  service 
to  customers 

•  Professional 
identity/perspective 

•  Competence 

•  Collegial  interaction 

From  Civilian  Studies 

•  Sex 

•  Time  in  career 

•  Organizational  norms 

•  Enhance 
personal/professional 
position 

•  Collegial  interaction 

•  Self-identity  in  their 
profession 

Benefits  of  Continuing  Education  to  the  Military 

The  Department  of  Defense  and  the  military  Services  have  studied  the  effects  and  value  of 
providing  continuing  education  to  both  enlistees  and  officers.  In  general,  this  literature  suggests 
that  continuing  education  programs  are  of  value  to  the  services.  Most  of  the  empirical 
evaluations  have  been  directed  at  three  criteria:  recruiting,  performance,  and  turnover3.  Although 


3 

It  should  be  recognized,  however,  that  these  three  criteria  are  likely  to  be  highly  interrelated,  and  could  lead  to 
spurious  conclusions.  For  example,  a  significant  relationship  between  CE  participation  and  promotion  may  be 
largely  explained  by  differences  in  retention  -  the  longer  you  stay  in  the  service  the  greater  the  chances  of 


12 


the  primary  goal  of  this  review  is  to  determine  the  impact  of  ACES  on  job  performance  and 
turnover,  we  begin  this  section  with  a  brief  discussion  of  the  potential  impact  of  ACES  on 
recruiting. 

Recruiting 

In  numerous  surveys,  new  recruits  and  soldiers  have  indicated  that  the  provision  of 
education  benefits  by  the  military  was  a  major  motivation  for  enlistment.  The  literature  in 
recruiting  and  education  benefits  focuses  almost  exclusively  on  the  recruiting  effects  of  the 
MGIB  and  the  ACF  “kickers,”  Although  we  did  not  review  this  body  of  literature  for  this  effort, 
numerous  studies  have  found  that  education  benefits  improve  recruiting  -  in  terms  of  both  the 
quantity  and  quality  of  recruits.  As  discussed  previously,  civilian  sector  studies  note  that  one  of 
the  major  reasons  that  employers  provide  CE  programs  for  their  employees  is  to  improve 
recruiting. 

Our  search  of  the  military  literature  did  not  produce  any  studies  that  looked  at  the  impact 
of  CE  programs  (for  active  duty  servicemembers)  on  recruiting.  The  paucity  of  research  in  this 
area  could  be  due  to  two  factors.  First,  little  information  is  available  concerning  whether 
potential  recruits  are  aware  of  programs  like  ACES.  That  is,  recruiters  may  be  promoting 
programs  like  the  G.I,  Bill  and  ACF,  but  providing  little  information  to  potential  recruits  on 
education  benefits  available  while  on  active  duty.  Second,  there  is  no  obvious  source  of  data  with 
which  to  conduct  analysis  linking  CE  programs  to  recruiting. 

Performance 

One  of  the  stated  goals  of  the  ACES  program  is  to  improve  the  effectiveness  of  the  force. 
Implicit  in  this  goal  is  that  participation  in  a  CE  program  will  enable  a  soldier  to  do  his  or  her  job 
more  effectively.  In  this  section  of  our  review,  we  look  at  empirical  investigations  of  the 
relationship  between  participation  in  CE  and  job  performance.  In  particular,  we  report  results 
from  six  studies  that  considered  performance  as  a  dependent  variable.  Descriptive  information 
for  each  of  the  studies  is  presented  in  Table  2. 

The  table  clearly  shows  the  variety  of  independent  variables  evaluated  in  these  studies. 

Of  the  five  studies,  for  example,  two  looked  at  participation  in  tuition  assistance  programs,  two 
looked  at  the  attainment  of  graduate  degrees,  and  one  looked  at  enrollment  in  the  Community 
College  of  the  Air  Force.  The  table  also  indicates  variability  in  the  operationalization  of 
performance,  with  the  most  common  operationalization  being  promotion.  While  performance 
and  promotion  could  be  considered  as  separate  outcome  variables,  we  believe  that  such  a 
distinction  would  be  artificial,  and  assume  that  promotion  is  a  direct  outcome  of  good 
performance.  On  the  other  hand,  using  promotion  as  a  measure  of  performance  may  exaggerate 
its  relationship  with  CE  participation,  because  that  participation  is  often  used  explicitly  in 


promotion.  We  will  provide  reviews  relevant  to  each  of  the  criteria,  and  not  attempt  to  address  such  mediated 
relationships. 
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Table  2 

Information  on  Studies  Examining  Promotion 


Study 

Service 

Education 

Programs 

Sample 

Dependent 

Variable 

Other  Variables  Controlled 
For 

Alley,  Mosley, 
Spivey,  Bolton,  & 
Mwambola  (1995) 

Air  Force 

Tuition 

Assistance 

Enlisted, 

Officer 

Ratings  of  how 
important  CE 
programs  are  for 
performance  and 
promotion 

Boesel  &  Johnson 
(1988) 

All 

Tuition 

Assistance 

Enlisted, 

Officer 

Promotion;  Self- 
rated 

expectations  of 
promotion 

Armed  Forces  Qualification  Test 
(AFQT)  category*,  enlistment 
period  ,  marital  status,  paygrade, 
race,  sex,  time  in  grade,  time 
remaining  in  enlistment  period*, 
total  active  federal  military  service 

Fuchs (1996) 

Navy 

Graduate 

Education 

Officer 

Executive 
officer  screen; 
Commanding 
officer  screen; 
Promotion  to  06 

academic  profile  code,  age  at 
commissioning,  commissioning 
source,  early  promotion,  marital 
status,  race/ethnicity,  sex,  technical 
preference  in  career  field,  type  of 
undergraduate  degree,  utilization  of 
graduate  education  at  promotion 
board 

Garcia  et  al.  (1998) 

Navy 

Tuition 

Assistance, 

PACE, 

Academic 

Skills 

Learning 

Centers 

Enlisted 

Promotions, 

demotions 

Education  at  accession,  vacancies, 

%  career  on  sea  duty,  AFQT  score, 
age,  sex,  race/ethnicity,  marital 
status,  accession  program, 
occupation 

Niemiec  (1987) 

Air  Force 

CCAF 

Enlisted 

Early  promotion 
vs.  late 
promotion 

Wielsma  (1996) 

Marines 

Graduate 

DegreesA 

Officer 

Average 

Performance 

Index; 

Promotion 

Average  performance  index  over 
career,  age,  sex,  race,  marital  status, 
occupational  community,  general 
classification  test  score,  composite 
ranking  at  the  basic  school, 
attendance  at  Naval  Academy, 
enrollment  in  ROTC,  participation 
in  OTC 

Notes:  *  indicates  a  variable  that  was  used  in  the  multivariate  analysis  of  enlisted  retention,  but  not  officer  retention. 
CCAF  =  Community  College  of  the  Air  Force.  A  In  this  study,  Marine  Corps  officers  with  graduate  degrees  were 
compared  to  officers  without  degrees. 
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making  promotion  decisions.  Finally,  the  table  indicates  three  of  the  studies  carried  out 
multivariate  analyses,  evaluating  the  relationship  between  participation  in  a  CE  program  and 
performance  while  controlling  for  other  explanatory  variables. 

The  results  of  these  studies  indicate  that  participation  in  CE  programs  leads  to  better 
performance.  Alley,  Mosley,  Spivey,  Bolton,  and  Mwambola  (1995)  surveyed  1,687  Air  Force 
officer  and  enlisted  professional  military  education  students  regarding  their  opinions  about  the 
tuition  assistance  and  off-duty  education  programs.  Results  indicated  that  38%  of  respondents 
believed  that  the  tuition  assistance  program  improved  officer  job  performance  and  66%  believed 
that  the  program  improved  enlisted  performance  (see  Alley  et  al.,  [1995]  Table  22).  When  asked 
more  generally  about  advanced  degrees,  24%  of  respondents  indicated  that  officers  with 
advanced  degrees  demonstrated  better  job  performance  than  officers  without  such  degrees. 
Similarly,  5 1  %  of  respondents  indicated  that  enlisted  personnel  with  advanced  degrees 
demonstrated  better  job  performance  than  those  without  such  degrees.  When  asked  about 
promotion,  67%  of  respondents  felt  that  having  an  advanced  degree  was  an  important  factor  in 
officer  promotion,  and  50%  felt  it  was  an  important  factor  in  enlisted  promotion  (see  Alley,  et 
al.,  [1995]  Table  27).  Interestingly,  however,  only  39%  indicated  that  having  an  advanced  degree 
should  be  considered  as  a  major  factor  in  officer  and  enlisted  promotion  decisions.  Thus,  there  is 
a  general  perception  that  possessing  an  advanced  degree  is  important  for  promotion,  but  less 
agreement  that  it  should  be  considered  for  promotion. 

Boesel  and  Johnson  (1988)  examined  the  relationship  between  participation  in  a  tuition 
assistance  program  and  promotion  in  a  sample  of  71,369  enlisted  and  officer  personnel  across 
three  of  the  military  Services.  Of  the  sample,  10,718  had  completed  a  tuition  assistance  course. 
Of  the  officers  in  the  sample,  46.8%  of  those  who  had  participated  in  a  tuition  assistance  course 
indicated  that  that  they  were  “Almost  Sure”  or  “Certain”  that  they  would  be  promoted,  whereas 
40.0%  of  officers  who  had  not  participated  in  such  a  course  gave  these  responses.4  These 
differences  were  even  larger  when  the  researchers  investigated  actual  promotion  records.  In 
particular,  Boesel  and  Johnson  examined  servicemember  promotion  records  over  an  18-month 
period.  They  found  that  53.1%  of  servicemembers  who  had  completed  a  tuition  assistance  course 
had  been  promoted  whereas  39.1%  of  servicemembers  who  had  not  taken  tuition  assistance 
courses  had  been  promoted  in  that  time  period. 

To  determine  whether  the  differences  in  promotion  could  be  attributable  to  factors  other 
than  participation  in  tuition  assistance  courses,  Boesel  and  Johnson  conducted  multivariate 
analyses  separately  on  enlisted  and  officer  samples.  The  evaluation  of  the  enlisted  sample 
indicated  that  the  relationship  between  tuition  assistance  participation  and  promotion  was  still 
strong  after  controlling  for  the  effects  of  sex,  race,  marital  status,  AFQT  score,  education, 
paygrade,  term  of  enlistment,  time  in  grade,  and  time  remaining  in  enlistment  period.  The 
multivariate  analysis  of  officer  promotion,  however,  indicated  no  relationship  between 
participation  in  the  tuition  assistance  program  and  promotion.  That  is,  the  univariate  relationship 
between  tuition  assistance  participation  and  promotion  was  fully  accounted  for  by  the  other 
variables. 


4  These  data  were  obtained  by  matching  the  database  to  the  1985  DoD  survey. 
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Niemiec  (1987)  examined  the  relationship  between  taking  courses  at  the  Community 
College  of  the  Air  Force  (CCAF)  and  promotion.  The  sample  consisted  of  3,001  individuals  who 
had  been  promoted  to  the  rank  of  Master  Sergeant.  A  median  split  technique  was  used  to  divide 
the  sample  into  two  groups,  those  who  attained  the  rank  early  and  those  who  attained  the  rank 
late.  The  results  indicated  a  modest  relationship  between  study  at  CCAF  and  promotion.  In 
particular,  80%  of  the  individuals  who  were  promoted  early  had  at  least  registered  for  courses  at 
CCAF,  whereas  only  72%  of  the  individuals  who  were  promoted  late  had  registered.  In  addition, 
20.5%  of  the  individuals  promoted  early  had  attained  a  degree,  whereas  only  9.9%  of  those 
promoted  late  had  attained  a  degree. 

Fuchs  (1996)  investigated  the  effects  of  participation  in  graduate  education  on  the 
promotion  of  field  grade  Naval  Officers.  His  sample  consisted  of  8,269  Naval  officers,  1,218 
who  had  participated  in  a  graduate  education  program.  The  study  shows,  overall,  that 
participation  in  fully-funded  graduate  education  has  a  positive  effect  on  three  different  officer 
career  progression  criteria,  including  (a)  executive  officer  screening,  (b)  commanding  officer 
screening,  and  (c)  promotion  to  0-6.  The  effect  of  graduate  education  in  all  three  types  of 
promotions  was  significant  and  positive.  In  the  executive  officer  screen,  for  example,  officers 
with  fully  funded  graduate  education  had  a  success  rate  of  69.5%  whereas  those  without  fully 
funded  graduate  education  had  a  success  rate  of  47.7%. 

Fuchs  also  found  that  officers  who  utilized  their  graduate  education  later  in  their  career 
progressions  had  a  greater  chance  for  promotion  than  were  those  who  used  their  graduate 
education  at  earlier  promotion  boards.  Finally,  officers  who  obtained  non-technical  graduate 
degrees  were  more  likely  to  be  promoted  than  were  those  officers  who  received  technical 
graduate  degrees.  Fuchs  speculates  that  this  is  because  non-technical  fields  of  study  may  be  more 
relevant  to  senior  management  duties. 

Research  by  Wielsma  (1996)  evaluated  performance  differences  between  Marine  Corps 
officers  with  graduate  degrees  and  those  without  graduate  degrees.  The  sample  consisted  of 
1,087  officers  who  entered  the  Marine  Corps  in  fiscal  year  1980,  78  of  whom  ultimately  obtained 
a  postgraduate  education.  A  unique  aspect  of  this  study  was  the  fact  that  it  included  a  measure  of 
on-the-job-performance,  the  average  performance  index.  Marine  officers  are  rated  on  a  fitness 
report  on  an  annual  basis.  The  performance  report,  which  includes  ratings  of  22  professional  and 
personal  characteristics,  is  scored  in  terms  of  three  dimensions:  performance,  qualities,  and 
overall  value  for  the  service.  The  average  performance  index  score  for  each  officer  in  the  study 
was  computed  as  the  average  score  on  the  performance  dimension  across  all  of  the  ratings  the 
individual  had  received  in  his  or  her  career.  Results  indicated  that  those  with  graduate  degrees 
had  significantly  higher  scores  on  the  average  performance  index  than  those  without  such 
degrees. 

Wielsma  also  evaluated  the  promotion  rates  among  those  officers  who  stayed  to  the  0-4 
promotion  point.  Results  indicated  that  while  79%  of  those  with  graduate  education  who  had 
stayed  to  the  promotion  point  were  promoted,  only  65%  of  the  officers  without  graduate 
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education  who  had  stayed  were  promoted,  A  multivariate  analysis5  indicated  that  graduate 
education  was  a  strong,  significant  predictor  of  promotion  in  an  initial  model  that  did  not  include 
the  performance  index  measure  or  general  classification  test  score  (a  marker  for  cognitive 
ability).  When  these  variables  were  added  to  the  model,  however,  the  magnitude  of  the  graduate 
education  variable  was  reduced,  but  it  remained  significant. 

Summary  of  findings  on  performance.  The  studies  reviewed  in  this  section  vary  widely  in 
terms  of  the  samples  used,  the  type  of  continuing  education  programs  evaluated,  and  the  ways  in 
which  they  define  performance.  Despite  this  variability,  however,  these  studies  all  generally 
indicate  that  continuing  education  programs  have  a  positive  effect  on  performance.  Although 
reduced  in  magnitude,  this  effect  appears  to  hold  up  in  multivariate  analyses  that  control  for 
other  potentially  explanatory  variables.  The  effect  would  also  appear  to  hold  for  both  officers 
and  enlisted  personnel. 

Because  most  of  the  studies  reviewed  used  promotion  as  a  measure  of  performance,  the 
results  should  be  interpreted  carefully  because  CE  participation  is  factored  into  the  promotion 
decision.  A  relationship  between  CE  and  promotion  may  merely  reflect  the  fact  that  participation 
in  civilian  education  can  give  a  servicemember  points  that  are  counted  in  determining  his  or  her 
eligibility  for  promotion.  Most  of  the  studies  described  in  this  review  mention  this  artifact,  but 
none  of  the  studies  use  statistical  techniques  to  isolate  the  impact  of  CE  participation  on 
promotions  independent  of  the  promotion  points  awarded  for  educational  attainment. 

Retention 

Turnover  is  costly  to  the  military.  To  replace  a  soldier  who  separates,  the  military  incurs 
recruiting  costs,  training  costs,  and  a  loss  of  experience  and  skills.  Furthermore,  when  soldiers 
separate,  the  military  incurs  permanent  change  in  station  (PCS)  costs,  administrative  costs  to 
outprocess  the  separating  soldier,  and  lost  productivity  during  the  time  the  soldier  is  transitioning 
out  of  the  military.  Our  review  of  the  literature  suggests  that  the  ACES  program  may  reduce 
turnover.  Enlisted  servicemembers  leave  the  military  for  many  reasons.  For  this  study,  we  look  at 
two  broad  categories  of  separations  for  enlisted  personnel:  (a)  failure  to  reenlist  and  the  end  of  a 
servicemember’ s  term  of  service,  and  (b)  attrition  during  a  term  of  service  (particularly  the  first 
term).  We  analyze  these  two  retention  outcomes  separately  because  attrition  and  reenlistment 
outcomes  typically  occur  at  different  stages  of  a  soldier’s  career.  In  addition,  the  opportunity  to 
participate  in  various  CE  programs  changes  throughout  a  soldier’s  career  so  the  impact  of 
participating  in  a  specific  CE  programs  may  vary  for  the  two  retention  outcomes. 

Reenlistment.  We  identified  seven  relevant  studies  on  the  impact  of  CE  programs  on 
retention  (Table  3).  Two  studies  are  of  enlistment  members  only;  two  studies  are  of  officers  only; 
and  three  studies  conduct  separate  analyses  for  both  enlisted  members  and  officers.  The  analyses' 
of  enlisted  members  use  reenlistment  outcomes  as  the  outcome  measure.  The  analyses  of  officers 
use  overall  retention  as  the  outcome  measure.  Four  of  the  studies  looked  explicitly  at 
participation  in  tuition  assistance  programs;  two  looked  at  participation  in  graduate  education; 


Wielsma  (1996)  presented  two  types  of  multivariate  analyses:  PROBIT  and  ordinary  least  squares.  Only  the  results 
from  the  PROBIT  analyses  are  reviewed  in  this  document. 
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Table  3 


Information  on  Studies  Examining  Retention. 


Study 

Service 

Education 

Programs 

Sample 

Dependent 

Variable 

Other  Variables 
Controlled  For 

Alley  et  al.  (1995) 

Air  Force 

Tuition 

Assistance 

Enlisted, 

Officer 

Rating  of 
relationship 
of  CE  to 
retention  and 
satisfaction 

Boesel  &  Johnson  (1988) 

All 

Tuition 

Assistance 

Enlisted, 

Officer 

Intention  to 

reenlist, 

reenlistment 

AFQT  category*,  enlistment 
period*,  marital  status,  paygrade, 
race,  sex,  time  in  grade,  time 
remaining  in  enlistment  period*, 
total  active  federal  military  service 

Brauchle  (1998) 

All 

Tuition 

Assistance 

Enlisted, 

Officer 

Intention  to 
reenlist 

race,  spouse’s  satisfaction  with 
his/her  education,  desire  to 
participate  in  off-duty  education  in 
the  previous  year,  education  level 
at  time  of  survey,  current 
enlistment*,  civilian  job 
opportunity,  job  satisfaction,  long¬ 
term  education  participation,  short¬ 
term  education  participation,  sex, 
marital  status,  satisfaction  with 
military  life,  pay  grade,  spouse’s 
satisfaction  with  military  life, 
education  level  at  entry,  total  active 
federal  military  service 

Burtzman  (1994) 

Navy 

FFGE 

Officer 

Annual 
retention  rate 

Garcia  et  aL  (1998) 

Navy 

Tuition 

Assistance, 

PACE, 

Academic 

Skills 

Learning 

Centers 

Enlisted 

Reenlistment 

Education  at  accession,  selective 
reenlistment  bonus  type,  pay  grade 
at  decision  point,  scheduled  for 
promotion,  sea  duty  or  next  tour 
ashore,  AFQT  score,  age,  sex, 
race/ethnicity,  number  of 
dependents,  marital  status, 
unemployment  rate,  occupation 

Simutis,  Ward,  Harman, 
Farr,  &  Kern  (1988) 

Army 

BSEP 

Enlisted 

Retention 
rate,  attrition 
rate 

Wielsma  (1996), 

Marines 

Graduate 

DegreesA 

Officer 

Staying  in 
service  to  0-4 
promotion 
point 

Average  performance  index  over 
career,  age,  sex,  race,  marital 
status,  occupational  community, 
composite  ranking  at  the  basic 
school,  attendance  at  Naval 

Academy,  enrollment  in  ROTC, 
participation  in  OTC 

Note:  *  indicates  a  variable  that  was  used  in  the  multivariate  analysis  of  enlisted  retention,  but  not  officer  retention. 
FFGE  =  Fully-funded  graduate  education.  BSEP  =  Basic  Skills  Education  Program.  A  in  this  study  Marine  Corps 
officers  with  graduate  degrees  were  compared  to  officers  without  degrees.  ' 
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and  one  focused  on  basic  skills  education.  Four  of  the  seven  studies  used  multivariate  regression 
analysis  to  isolate  the  impact  of  CE  participation  on  retention  while  controlling  for  other 
explanatory  variables  that  were  hypothesized  to  affect  retention. 

In  their  survey  of  officer  and  enlisted  professional  military  education  students.  Alley  et 
al.  (1995)  asked  several  questions  concerning  the  perceived  influences  of  the  tuition  assistance 
program  on  retention.  Results  indicated  that  65%  of  respondents  agreed  (rated  the  item  as 
“Strongly  Agree”  or  “Agree”)  that  a  major  reason  enlisted  people  stay  in  the  military  is  because 
of  the  educational  opportunities  (compared  to  10%  that  indicated  “Disagree”  or  “Strongly 
Disagree”).  The  trend  results  were  somewhat  different  for  officers.  That  is,  only  20%  of 
respondents  agreed  that  officers  stay  in  the  military  because  of  the  educational  opportunities 
(compared  to  33%  who  disagreed). 

In  terms  of  job  satisfaction,  which  some  argue  is  connected  to  retention,  respondents  felt 
that  tuition  assistance  improved  the  job  satisfaction  of  both  officers  and  enlisted  personnel. 
Specifically,  39%  agreed  that  tuition  assistance  improved  officer  job  satisfaction  (11% 
disagreed),  and  68%  agreed  that  it  improved  enlisted  satisfaction  (6%  disagreed). 

Boesel  and  Johnson  (1988)  also  examined  the  relationship  between  participation  in  a 
tuition  assistance  program  and  retention.  Based  on  data  from  a  1985  DoD  survey,  they  found  that 
13.4%  of  the  people  in  their  sample  that  had  participated  in  tuition  assistance  planned  on  leaving 
the  service  at  the  end  of  their  current  commitment.  In  contrast,  they  found  that  23.6%  of  people 
in  their  sample  who  had  not  participated  in  tuition  assistance  planned  on  leaving  the  service. 

This  difference  was  even  more  dramatic  when  actual  retention  was  evaluated.  Of  the 
people  who  had  participated  in  tuition  assistance,  18,6%  had  left  the  military  in  the  18-month 
time  period  examined  by  Boesel  and  Johnson,  The  failure  to  reenlist  rate  was  much  higher 
(35.8%)  among  people  who  did  not  participate  in  TA.  The  strong  significant  relationship 
between  participation  in  TA  and  retention  was  found  among  both  enlisted  and  officer  personnel 
even  after  controlling  for  the  effects  of  other  explanatory  variables. 

A  study  by  Brauchle  (1998)  was  designed  as  a  replication  and  extension  of  the  Boesel 
and  Johnson  (1988)  study.  Brauchle  used  data  from  a  1992  DoD  survey  to  evaluate  the 
relationship  between  ever  having  participated  in  CE  and  self-reported  intention  to  reenlist. 6  The 
correlation  between  the  CE  participation  measure  and  the  intention  to  reenlist  was  found  to  be 
weak  (accounting  for  only  3%  of  the  variation  in  the  intention  to  reenlist),  but  statistically 
significant.  The  results  of  multivariate  analysis  were  similar;  long-term  participation  continued  to 
be  a  significant  predictor  of  the  intention  to  reenlist,  but  the  amount  of  variance  accounted  for  by 
this  variable  was  relatively  small. 


6  This  measure  of  CE  participation  was  created  based  on  survey  responses  to  questions  concerning  education  level  at 
time  of  entry  (into  the  military)  and  the  education  level  possessed  at  the  time  the  survey  was  completed.  The 
participation  measure  was  defined  as  an  increase  in  education  level  (from  entry  to  the  time  of  the  survey),  with  the 
restriction  that  the  individual  had  completed  at  least  “some  college.” 
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In  an  extension  of  the  Boesel  and  Johnson  (1988)  analysis,  Brauchle  included  additional 
measures  in  his  multivariate  analyses,  including  job  satisfaction,  satisfaction  with  the  military 
way  of  life,  and  civilian  employment  prospects.  He  found  that  this  more  complex  model 
explained  almost  41%  of  reenlistment  behavior  compared  to  the  replicated  model  that  explained 
only  25%.  The  best  predictor  of  reenlistment  intention  in  the  more  complex  model  was 
satisfaction  with  military  life,  which  explained  26%  of  reenlistment  behavior.  In  this  model,  ever 
having  participated  in  off-duty  education  accounted  for  just  under  8%  of  the  variation  in  intent  to 
reenlist.  Though  off-duty  education  participation  does  not  account  for  a  large  percentage  of  a 
person’s  reasons  to  reenlist,  the  author  nevertheless  concludes  that  retention  rates  are  higher 
among  those  who  do  and  who  want  to  participate  in  off-duty  education  (even  if  they  don’t 
participate),  controlling  for  education  level. 

Brauchle  notes  that  members  with  longer  service  are  both  more  likely  to  have 
participated  in  a  CE  program  during  their  military  career  and  are  more  likely  to  reenlist. 
Consequently,  he  assessed  a  short-term  measure  of  participation  in  a  CE  program — i.e.,  having 
participated  during  the  previous  year.  Brauchle  reports  a  very  small,  but  significant  negative 
relationship  between  this  variable  and  the  intention  to  reenlist.  He  speculates  that  those  who  plan 
to  leave  the  service  take  advantage  of  the  opportunity  to  receive  financial  assistance  and  prepare 
for  the  civilian  job  market  in  greater  numbers  than  do  those  who  intend  to  remain  in  service. 
These  results  underscore  the  importance  of  examining  as  much  of  a  servicemember’s  history  as 
possible  in  evaluating  the  relationship  between  CE  participation  and  reenlistment. 

The  primary  focus  of  research  conducted  by  Brutzman  (1994)  was  toward  an  evaluation 
of  the  utilization,  defined  as  serving  a  tour  in  a  billet  related  to  the  subject  area  of  the  graduate 
education,  of  Navy  officer  personnel  who  received  fully-funded  graduate  education  (FFGE). 
While  utilization  is  unrelated  to  the  purposes  of  the  present  review,  she  also  examined  the 
relationship  between  FFGE  and  retention.  Using  a  longitudinal  database,  she  compared  the 
percentage  of  FFGE  officers  who  left  the  Navy  to  the  percentage  of  non-FFGE  officers  who  left 
the  Navy  for  each  of  the  years  1981  to  1993  (with  the  exception  of  1983).  This  comparison 
indicated  that  the  percentage  of  FFGE  officers  leaving  the  Navy  was  lower  in  every  year.  Across 
the  years,  an  average  of  4.8%  of  all  FFGE  officers  left  per  year  whereas  an  average  of  1 1.2%  of 
non-FFGE  officers  left.  She  also  indicates  that  “73.1%  of  all  FFGE  officers  remained  in  the 
service  past  their  commitment”  (p.53),  which  is  a  retention  rate  nearly  double  that  in  the  non- 
FFGE  groups. 

In  his  comparison  of  Marine  Corps  officers  with  and  without  postgraduate  education, 
Wielsma  (1996)  also  considered  the  effects  on  retention.  In  this  study,  retention  was  defined  as 
staying  in  the  service  to  the  0-4  promotion  point.  It  was  found  officers  choosing  to  stay  in  the 
Marines  are  more  likely  to  have  obtained  a  postgraduate  education.  Although  only  7%  (n  =  78) 
of  the  sample  had  graduate  degrees,  15%  (n  =  67)  of  the  people  who  stayed  to  the  0-4  promotion 
point  had  graduate  degrees.  Looking  at  this  analysis  differently,  83%  of  those  with  graduate 
education  stayed  to  the  0-4  point.  This  percentage  stands  in  dramatic  contrast  to  the  38%  of 
those  without  graduate  education  who  stayed.  Wielsma  also  conducted  a  multivariate  analysis  to 
evaluate  this  effect.  In  this  analysis,  graduate  education  was  a  strong,  significant  predictor  of 
retention  in  an  initial  model  that  did  not  include  the  performance  index  measure  (the  general 
classification  test  score  variable  was  not  included  in  this  analysis).  When  the  measure  of  on-the- 
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job  performance  was  added  to  the  model  the  magnitude  of  the  graduate  education  variable  was 
reduced,  but  it  remained  statistically  significant. 

Research  by  ARI  (Simutis,  Ward,  Harman,  Farr,  &  Kern,  1988)  indicates  that  BSEP  also 
increase  retention  rates.  For  example,  they  found  that  a  sample  of  3,271  BSEP  graduates  had 
lower  attrition  rates  (3.4%  vs.  34.6%)  and  higher  reenlistment  rates  (37.9%  vs.  1 1.0%)  than  a 
comparison  group  (n  =  3,328). 

The  findings  of  this  literature  review  support  the  hypothesis  that  CE  programs  improve 
retention.  People  who  complete  some  form  of  CE  program  tend  to  stay  in  the  services  longer 
than  those  who  do  not.  Multivariate  analyses  have  also  indicated  significant  positive 
relationships  between  CE  participation  and  retention  when  other  variables  have  been  taken  into 
account.  The  small  number  of  studies  on  enlisted  servicemembers  and  data  and  methodological 
problems  with  the  studies  we  reviewed  do  not  allow  us  to  estimate  the  size  of  the  impact  on 
retention. 

Attrition.  Attrition  is  a  subset  of  total  separations  and  is  an  issue  that  pertains  mainly  to 
enlisted  personnel  in  their  first  term  of  service.  Although  numerous  studies  have  analyzed  the 
causes  of  attrition  in  the  military,  to  our  knowledge  the  study  by  Simutus  et  al.  (1988)  described 
above  is  the  only  one  that  investigated  the  effect  a  CE  program  may  have  on  reducing  attrition. 
One  reason  for  the  paucity  of  research  in  this  area  may  be  that  a  large  percentage  of  attrition 
occurs  early  in  the  initial  enlistment.  Consequently,  many  soldiers  who  separate  early  have  not 
had  the  opportunity  to  become  informed  about,  or  participate  in,  the  military’s  CE  program.  The 
BSEP  program  evaluated  by  Simutus  et  al.  (1988)  is  one  that  is  available  to  a  soldier  early  in  his 
or  her  career,  and  consequently  would  be  more  likely  to  reduce  attrition. 

In  the  remainder  of  this  section  we  review  the  general  literature  on  attrition  to  provide 
information  on  the  data  and  methods  used  in  previous  studies  to  model  attrition.  Attrition  can 
occur  for  numerous  reasons,  some  of  which  are  beyond  the  military’s  control.  Consequently,  it  is 
useful  to  construct  two  working  definitions  of  attrition — “voluntary”  attrition  and  “involuntary” 
attrition.  Voluntary  attrition  is  defined  as  those  separations  that  are  the  result  of  the  soldier’s 
actions  (e.g.,  the  decision  to  leave,  poor  performance  or  unacceptable  behavior).  Involuntary 
attrition  is  defined  as  those  separations  that  are  not  the  result  of  choice  (e.g.,  death,  and  medical 
and  psychological  disability).  While  most  reasons  for  separation  can  be  classified  unambiguously 
as  either  voluntary  attrition  or  involuntary  attrition,  the  classification  is  not  straightforward,  and 
may  be  arbitrary,  in  many  cases.7 

The  main  reason  that  an  analysis  should  distinguish  between  voluntary  and  involuntary 
attrition  is  to  build  a  causal  model  of  the  attrition  process  that  can  accurately  capture  the 
relationship  between  attrition  and  its  explanatory  variables.  Previous  research  has  shown  that  this 


7  There  is  some  concern  whether  the  Army’s  separation  data  are  sufficiently  reliable  to  separate  attrition  into 
meaningful  categories.  For  example,  a  soldier  might  receive  a  medical  discharge  when  the  true  reason  for  separating 
was  not  medical  related.  To  the  extent  possible,  involuntary  separations  (e.g,,  death)  that  can  be  identified  should  be 
excluded  from  the  analysis. 
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relationship  differs  by  reason  for  separation  (see,  for  example,  Klein  and  Martin,  1991).  Most 
research  has  focused  on  the  issue  of  voluntary  attrition. 

Hogan,  Smith  and  Sylwester  (1991)  conducted  a  study  that  investigates  the  impact  of  the 
ACF  on  both  attrition  and  reenlistment.  They  found  that  supplemental  education  benefits  have 
only  a  small,  statistically  nonsignificant  effect  on  contract  completion.  Unlike  the  BSEP 
program,  which  is  often  used  by  soldiers  early  in  their  career,  the  ACF  is  primarily  used  by 
soldiers  after  they  separate.  Consequently,  it  is  not  surprising  that  Hogan  et  al.  found  a  much 
smaller  effect  for  that  program  than  Simutus  et  al.  (1988)  did  for  BSEP. 

Laurence,  Naughton  and  Harris  (1995)  reviewed  the  attrition  literature  and  discussed  the 
known  and  suggested  causes  of  first-term  attrition.  Below,  we  summarize  the  explanatory 
variables  used  in  previous  analyses  of  attrition. 

•  Contract  length.  Hogan  (1979)  shows  that  longer  contract  length  is  positively  correlated 
with  attrition.  However,  estimation  of  the  magnitude  of  the  theoretical  relationship 
between  contract  length  and  the  probability  of  separating  prior  to  contract  completion  is 
complicated  by  the  likelihood  that  soldiers  with  a  higher  taste  for  military  life — and  thus 
at  lower  risk  of  attrition — may  be  more  likely  to  choose  contracts  of  greater  length. 

•  Education  (as  measured  by  years  completed,  diploma,  and  GED).  Many  studies  have 
found  that  having  a  high  school  diploma  is  the  best  single  predictor  of  completing  the 
first-term  enlistment.  However,  the  reason  why  high  school  graduates  are  less  likely  than 
non-graduates  to  separate  early  is  unclear.8 

•  Mental  ability.  Enlistees  with  higher  AFQT  scores  are  less  likely  to  separate  early  than 
those  with  lower  scores  (see,  for  example.  Flyer  and  Elster,  1983;  Laurence,  1984,  1987; 
Klein  and  Martin,  1991).  In  addition,  AFQT  has  been  found  to  be  a  better  predictor  of 
attrition  among  high  school  graduates  and  for  whites  versus  blacks  (Elster  and  Flyer, 
1982).  In  this  analysis,  average  grade  in  high  school  provides  an  additional  proxy  for 
mental  ability  (although  grades  are  a  function  of  both  ability  and  effort). 

•  Military  occupation  and  skills.  Past  studies  have  found  differences  in  attrition  rates 
between  occupational  specialties  in  the  military  (e.g,,  Fernandez,  1985;  Finstuen  &  Alley, 
1983;  and  Rosenthal  &  Laurence,  1988).  Reasons  may  be  that  some  jobs  are  more 
arduous  or  onerous  than  others.  Also,  in  some  occupations  soldiers  are  learning  skills  that 
are  more  marketable  in  the  civilian  workforce. 

•  Race/ethnicity.  The  literature  shows  mixed  findings  on  the  relationship  between 
race/ethnicity  and  attrition.  Cooke  and  Quester  (1988)  find  that  relative  to  members  of 


8  Plausible  explanations  are  that  ability  and  personal  skills  that  contribute  to  a  successful  graduation  are  the  same 
factors  that  contribute  to  the  successful  completion  of  one’s  enlistment  contract.  Consequently,  a  high  school 
diploma  not  only  represents  a  level  of  academic  success,  but  also  represents  unobservable  characteristics  such  as 
ability  and  degree  of  discipline.  Laurence  (1987)  found  that  attrition  rates  of  soldiers  with  a  GED  more  closely 
resemble  attrition  rates  of  non-high  school  graduates  than  of  graduates. 
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racial  or  ethnic  minority  groups,  whites  are  more  likely  to  be  discharged  for 
administrative  reasons  and  less  likely  to  be  discharged  for  disciplinary  actions.  Klein  and 
Martin  (1991)  find  that  all  else  being  equal,  white  recruits  are  more  likely  than  their  black 
counterparts  to  separate  early  both  for  medical  and  adverse  reasons. 

•  Sex.  Various  studies  have  found  that  women  are  more  likely  to  separate  early  than  men 
(e.g.,  Flyer  and  Elster,  1983).  Compared  to  men,  women  are  more  likely  to  separate  for 
medical  reasons  (often  for  pregnancy)  and  less  likely  to  separate  for  disciplinary-related 
actions. 

•  Supplemental  education  benefits.  Hogan,  Smith  and  Sylwester  (1991)  find  that 
supplemental  educational  benefits  offered  under  the  Army  College  Fund  have  a  small, 
negative  effect  on  attrition.  However,  they  find  that  the  relationship  is  not  statistically 
different  from  zero. 

•  Age.  Past  studies  have  shown  that  the  relationship  between  age  and  attrition  is  not 
especially  strong,  although  there  is  some  evidence  that  younger  soldiers  are  more  likely 
than  their  older  counterparts  to  separate  because  of  behavioral  problems  and  older 
soldiers  are  more  likely  than  younger  soldiers  to  separate  for  medical  reasons. 

•  Marital  status  and  number  of  dependents.  Both  marital  status  and  number  of  dependents 
can  vary  across  soldiers  and  over  time.  Past  studies  are  fairly  consistent  in  finding  that 
married  soldiers  are  more  likely  to  separate  early  than  single  soldiers,  although  the 
relationship  may  be  weak  (Klein  and  Martin,  1991).  This  pattern  holds  for  both  male  and 
female  soldiers.  Little  research  has  been  conducted  to  determine  whether  attrition  is 
correlated  with  having  dependents  or  with  the  number  of  dependents. 

•  Economic  conditions.  The  ratio  of  military  to  civilian  pay  and  the  unemployment  rate  are 
two  possible  explanatory  variables  to  control  for  economic  conditions.  Kleinman  and 
Zuhoski  (1980)  estimate  the  effect  of  pay  and  other  determinants  on  Navy  pilot  attrition. 
They  find  that  pilot  attrition  increases  as  the  pay  of  civilian  pilots  increases  relative  to 
military  pay. 

Many  of  the  studies  we  reviewed  model  the  interaction  of  the  explanatory  variables  described 
above.  For  example,  Klein  and  Martin  model  the  interaction  of  race  and  AFQT  score,  and  the 
interaction  of  race/ethnicity  and  age. 

Methodological  Issues  and  Data  Limitations 

The  empirical  studies  that  we  reviewed  encountered  numerous  methodological  issues  and 
data  limitations  that  are  relevant  to  this  study.  Failure  to  address  these  issues  could  potentially 
reduce  the  reliability  of  the  findings  and  add  bias  to  the  evaluation.  As  discussed  previously, 
individuals  who  participate  in  employer-sponsored  education  and  training  programs  are  chosen 
either  through  self-selection,  or  by  the  employer.  There  is  no  random  assignment.  Furthermore, 
many  of  the  same  factors — such  as  ability  and  motivation — that  help  determine  program 
participation  also  influence  the  job  performance  and  retention  outcomes  that  we  desire  to 
analyze.  To  obtain  unbiased  estimates  of  the  impact  of  CE  program  participation  on  the 
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outcomes  of  interest,  one  must  construct  an  experimental  design  that  controls  for  the  non-random 
nature  of  selection  for  program  participation. 

Isolating  the  value  added  by  CE  program  participation  is  made  difficult  by  the 
confounding  relationship  between  the  outcomes  of  interest  (i.e.,  recruiting,  performance,  and 
retention)  and  the  attributes  of  individuals  in  the  sample.  Methodological  issues  and  data 
limitations  further  complicate  the  analysis.  In  this  section  we  describe  the  methodological  and 
data  issues  encountered  in  the  empirical  literature.  We  give  a  brief  description  of  the  each  issue 
and  describe  the  techniques  used  in  past  studies  to  address  these  issues.  These  issues  are 
(a)  evaluation  design  and  selection  bias,  (b)  data  limitations,  and  (c)  sampling  issues. 

Evaluation  Design  and  Selection  Bias 

The  studies  we  reviewed  all  use  a  retrospective  evaluation  design  where  the  education 
programs  were  evaluated  using  historical  data  and  where  the  evaluator  had  little  or  no  input  into 
the  process  by  which  individuals  were  selected  to  participate  in  the  education  program  evaluated. 
The  optimal  experimental  design,  in  terms  of  obtaining  unbiased  findings,  would  be  a 
“controlled”  experiment  in  which  members  of  the  relevant  population  (e.g.,  soldiers)  were 
randomly  assigned  to  a  test  group  (e.g.,  individuals  eligible  to  participate  in  the  CE  program)  or 
to  a  control  group  (e.g.,  individuals  not  eligible  to  participate).  Then,  data  on  the  outcome  of 
interest  (i.e.,  recruiting,  performance,  and  retention)  would  be  collected  over  time  to  determine  if 
there  are  systematic  and  significant  differences  in  the  outcomes  of  individuals  in  the  test  and 
control  groups. 

Because  virtually  every  soldier  is  eligible  to  participate  in  the  major  CE  programs  under 
ACES  and  because  participation  in  the  CE  programs  reviewed  is  voluntary,  such  an  ideal 
“experiment”  is  not  possible.  Controlling  for  the  voluntary  nature  of  program  participation  is 
vital  to  isolating  the  CE  programs’  impact  on  the  outcomes  of  interest.  For  example,  Fuchs 
(1996)  found  that  Naval  officers  with  a  stronger  academic  background  and  more  favorable 
performance  ratings  early  in  their  career  were  more  likely  to  pursue  a  graduate  education  and  had 
a  higher  probability  of  promotion  to  commander. 

Because  a  controlled  experiment  with  random  assignment  generally  is  not  feasible, 
researchers  have  used  “quasi-experimental”  evaluation  designs  to  mitigate  the  effect  of  selection 
bias.  A  quasi-experimental  design  controls  for  factors  that  affect  both  assignment  to  the  test 
group  (i.e.,  CE  program  participation)  and  the  outcomes  being  analyzed.  The  two  main 
approaches  to  conduct  a  quasi-experimental  design  are  multivariate  regression  analysis  and 
matched-pairs  analysis.  The  empirical  studies  that  we  reviewed  use  the  former  approach 
exclusively. 

Wielsma  (1996)  used  a  multivariate  regression  analysis  to  determine  how  attending 
graduate  school  affects  performance  and  retention  of  USMC  officers.  Talaga  (1994)  estimated 
three  regression  models  to  determine  the  impact  of  graduate  education  on  three  measures  of 
performance  for  naval  surface  warfare  officers.  Fuchs  (1996)  estimated  a  regression  model  to 
analyze  the  impact  of  participating  in  a  graduate  education  program  on  the  promotion  of  Naval 
officers.  Garcia  et  al.  (1998)  estimated  a  series  of  regression  models  to  determine  how 
participation  in  the  Navy’s  Voluntary  Education  (VOLED)  program  affected  promotions  and 
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retention  of  sailors.  Boesel  and  Johnson  (1988)  used  the  regression  model  approach  to  determine 
how  participation  in  a  DoD  Tuition  Assistance  program  affected  retention  in  the  military  and 
promotions  of  enlisted  soldiers.  The  empirical  rigor  of  these  studies  and  the  ability  to  generalize 
their  findings  to  ACES  varies  from  study  to  study,  but  the  literature  suggests  that  the  evaluation 
of  some  ACES  programs,  particularly  TA  and  FAST,  will  find  a  modest,  positive  effect  of  ACES 
program  participation  on  soldier  retention  and  promotions.9  However,  the  effects  of  many  ACES 
programs,  such  as  MOS  improvement  training  and  NCO  leadership  training,  have  not  been 
assessed  by  previous  evaluations. 

The  purpose  of  using  a  multivariate  regression  is  to  isolate  the  effect  of  each  explanatory 
variable  on  the  dependent  variable.  Because  ACES  participation  is  voluntary,  and  because  many 
of  the  factors  that  determine  program  participation  are  also  predictors  of  performance  and 
retention,  the  estimates  from  the  regression  model  may  be  biased  unless  one  controls  for  self¬ 
selection.  Several  approaches  have  been  suggested  in  the  literature  to  mitigate  the  problem  of 
selection  bias.  These  approaches  are  not  necessarily  mutually  exclusive. 

The  first  approach  is  to  estimate  a  regression  model  that  contains  all  observable  soldier 
characteristics  that  help  determine  program  participation  (i.e.,  control  variables)  and  explanatory 
variables  that  affect  the  outcome  of  interest.  0  Inclusion  of  the  control  variables  help  minimize 
the  problem  of  “selection”  bias,  while  inclusion  of  the  explanatory  variables  help  minimize  the 
problem  of  “omitted”  variable  bias.  Factors  such  as  pay,  bonuses,  and  MOS  that  may  affect  the 
outcomes  of  interest  should  be  included  in  the  regression  analysis.  Even  though  these  factors 
may  be  uncorrelated  with  program  participation,  including  them  in  the  model  will  reduce  the 
residual  variance  and  thus  increase  the  precision  of  the  estimated  program  effect.  To  the  extent 
that  one  can  successfully  include  the  variables  that  are  correlated  with  participation  and  that  also 
affect  retention  and  promotions,  one  will  obtain  an  unbiased  estimate  of  the  program  effect, 
using  participation  as  the  “treatment”  indicator.  However,  if  one  omits  some  variables  that  are 
correlated  with  participation  and  that  affect  outcomes,  the  estimated  program  effect  may  still 
suffer  from  selection  bias. 

The  second  approach  requires  that  two  regression  models  be  estimated.  This  approach  is 
sometimes  referred  to  as  the  “Heckman  two-step  procedure.”  The  first  step  is  to  estimate  a  probit 
model  to  predict  the  probability  of  program  participation.  This  probability  is  manipulated  to  form 
a  ratio,  known  as  the  “Inverse  Mills  Ratio,”  that  is  used  as  a  control  variable  in  the  second 
regression.  The  second  regression  contains  all  the  explanatory  variables  hypothesized  to  affect 
the  dependent  variable,  in  addition  to  the  Inverse  Mills  Ratio.  This  approach  was  used  by  Boesel 
and  Johnson  (1988),  Garcia  et  al.  (1998),  and  Wielsma  (1996). 


9  Although  the  retention  effect  is  likely  to  be  small,  even  a  small  increase  in  retention  may  translate  into  large  dollar 
savings  to  the  Army  in  terms  of  reduced  recruiting  and  training  costs. 

10  In  the  econometrics  literature  on  program  evaluation,  this  is  sometimes  called  “selection  on  observables”  in  that 
observable,  measurable  factors  affecting  both  participation  and  outcomes  are  explicitly  controlled  for  by  including 
them  in  the  multivariate  estimation  equation. 
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Boesel  and  Johnson  (1988)  estimated  a  probit  model  of  participating  in  a  TA  program. 
Then,  the  authors  estimated  a  logit  model  to  determine  the  relationship  between  remaining  in  the 
military  during  the  period  July  1986  to  December  1987  and  independent  variables— including 
the  Inverse  Mills  ratio.  They  also  developed  a  similar  model  to  estimate  the  relationship  between 
promotions  during  the  period  July  1986  to  December  1987  and  these  independent  variables.11 

Garcia  et  al.  (1998)  used  the  Heckman  two-step  procedure  to  control  for  voluntary 
participation  in  the  Navy’s  VOLED  program  in  their  study  of  the  impact  of  VOLED 
participation  on  reenlistment  and  promotions  of  Navy  enlisted  sailors.  The  authors  estimated 
logit  models  to  predict  retention  and  promotion  as  a  function  of  VOLED  program  participation 
and  various  sailor  attributes.  Wielsma  (1996)  used  the  Heckman  procedure  to  control  for 
selection  bias  in  his  study  of  the  effects  of  graduate  education  on  promotions  of  USMC  officers. 

A  third  approach  to  control  for  selection  bias  is  useful  if  there  are  repeated  observations 
on  the  “outcome”  for  the  individual  soldier,  but  variation  over  time  in  the  soldier’s  participation. 
In  this  case,  the  soldier  serves  as  his  or  her  own  “control”  and  one  examines  changes  in  outcomes 
before  and  after  program  participation.  One  example  is  when  enlisted  naval  personnel  not 
qualified  for  A-school  retake  the  AFQT  in  an  attempt  to  qualify  for  A-school.  None  of  the 
studies  that  we  reviewed  use  this  approach. 

A  fourth  approach  is  to  examine  ACES  program  history  to  determine  whether  there  is 
variation  over  time  in  when  a  program  is  offered  or  in  access  to  programs.  These  would 
potentially  constitute  “natural  experiments.”  Program  effects  would  be  measured  based  not 
necessarily  on  actual  participation,  but  on  the  opportunity  to  participate.  The  measured  impact 
would  be  the  impact  of  the  program  on  the  soldiers  who  had  access  to  the  program  compared  to 
soldiers  who  did  not,  after  controlling  for  other  differences  between  the  two  groups  that  may 
potentially  affect  outcomes.  Because  the  individual’s  actual  choice  to  participate  or  not  is  not 
used  as  the  treatment  indicator,  potential  self-selection  bias  is  reduced.  None  of  the  studies  that 
we  reviewed  use  this  approach. 

An  alternative  to  the  multivariate  regression  approach  to  design  a  quasi-experimental 
evaluation  is  a  matched-pairs  analysis.  For  this  approach,  the  researcher  first  identifies  a  sample 
of  individuals  who  participated  in  the  program  and  thus  self-selected  into  the  test  group.  To  form 
a  control  group,  the  researcher  identifies  a  “match”  for  each  individual  in  the  test  group  using  the 
attributes  of  the  individual  to  make  the  match.  A  major  problem  with  this  approach  is  that 
matching  is  difficult,  and  an  inaccurate  matching  scheme  will  lead  to  inaccurate  results. 


11  One  can  make  several  criticisms  of  this  study  by  Boesel  and  Johnson.  First,  the  authors  pooled  data  on  soldiers  of 
different  grades  and  different  enlistments  instead  of,  for  example,  estimating  different  regression  models  for  soldiers 
in  their  first,  second,  or  third  enlistment.  The  relationship  between  the  independent  variables  (e.g.,  TA  participation) 
and  retention  likely  are  very  different  for  soldiers  in  their  first  enlistment  versus  soldiers  in  their  second  or  third 
enlistment.  Second,  the  authors  used  a  continuous  variable  for  pay  grade.  Thus,  the  relationship  between  grade  and 
the  probability  of  remaining  in  the  military  was  assumed  to  be  constant  over  all  grades  (e.g.,  E-l  versus  E-2,  and  E-2 
versus  E-3,  etc.).  The  same  criticism  holds  for  the  analysis  of  promotions.  Third,  the  authors  did  not  consider 
whether  a  soldier’s  enlistment  ended  during  the  July  1986  to  December  1987  time  period.  Consequently,  the  model 
likely  overpredicts  the  probability  of  remaining  in  the  Services.  Whether  this  misspecification  biased  the  coefficients 
on  the  independent  variables  is  unknown. 
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What  factors  have  been  shown  to  affect  the  propensity  of  soldiers  to  participate  in  a 
voluntary  education  program  and  also  are  hypothesized  to  affect  retention  and  job  performance? 
As  discussed  previously,  Boesel  and  Johnson  (1988)  found  that  AFQT  score,  level  of  education, 
race,  sex,  and  rank  all  were  correlated  with  TA  program  participation.  In  particular,  the  soldier  is 
more  likely  to  participate  in  the  TA  program  if  the  soldier  has  a  higher  AFQT  score,  has  a  higher 
level  of  education  (up  to  having  a  college  degree),  is  black,  is  female,  and  is  a  sergeant  (E5 
through  E7). 

Garcia  et  al.  (1998)  found  that  the  probability  of  participating  in  the  Navy’s  VOLED 
program  was  statistically  higher  if  the  sailor  was  female;  Hispanic  or  Asian  Pacific  Islander;  and 
was  in  an  administration,  aviation  supply,  or  medical  career.  The  participation  probability 
decreased  with  sailor  age  at  time  of  accession,  if  the  sailor  had  been  demoted,  and  percent  of 
career  on  sea  duty.  Talaga  (1994)  estimated  a  model  to  predict  enrollment  in  the  Navy’s 
postgraduate  school.  Positive  and  statistically  significant  predictors  of  program  participation 
included  various  measures  of  academic  ability  (i.e.,  undergraduate  grade  point  average  and  a 
measure  of  math  skills  performance),  measures  of  job  performance  (i.e.,  recommendation  for 
early  promotion,  qualification  for  Surface  Warfare  or  Engineering  Officer  of  the  Watch  before 
the  0-3  promotion  board,  or  qualification  for  Tactical  Action  Officer),  and  number  of  curricula 
for  which  the  officer  was  eligible. 

Data  Limitations 

The  studies  we  reviewed  encountered  several  data  limitations  that  are  relevant  to  an 
evaluation  of  the  ACES  program.  These  issues  are  sample  attrition,  censoring,  and  measurement 
error. 


Sample  Attrition.  Sample  attrition  occurs  when  members  leave  the  sample  before  the  end 
of  the  data  collection  period.  Failure  to  control  for  sample  attrition  may  bias  the  findings.  Below 
we  provide  a  brief  description  of  the  issue  as  discussed  in  the  literature. 

Consider  the  following  example  that  illustrates  how  sample  attrition  may  affect  the 
evaluation  of  ACES.  Suppose  one  wishes  to  design  an  evaluation  of  the  effect  of  CE  program 
participation  on  promotions.  The  researcher  will  collect  information  on  a  sample  of  soldiers  who 
participate  in  the  program  (i.e.,  the  test  group)  and  soldiers  who  did  not  participate  In  the 
program  (i.e.,  the  control  group).  Then,  the  researcher  will  determine  if  soldiers  in  the  test  group 
were  more  likely  to  be  promoted  during  a  given  period  of  time  (e.g,,  within  two  years  after 
participating  in  the  education  program).  Some  soldiers,  however,  may  leave  the  military  before 
the  end  of  the  data  collection  period.  Thus,  one  never  observes  whether  the  soldier  was 
promoted.  If  the  reason  for  leaving  is  related  either  to  participation  in  the  CE  program  or  to  the 
outcome  of  interest,  then  sample  attrition  may  bias  the  findings. 

In  this  example,  if  a  soldier  thinks  he  or  she  will  likely  be  promoted,  then  the  soldier  may 
decide  to  reenlist.  Alternatively,  if  the  soldier  thinks  he  or  she  will  not  be  promoted,  then  the 
soldier  may  decide  not  to  reenlist.  If  ACES  participation  increases  the  likelihood  of  promotion, 
then  failure  to  control  for  this  sample  attrition  would  cause  one  to  overestimate  the  impact  of 
ACES  on  promotions.  In  this  hypothetical  scenario,  soldiers  who  do  not  participate  in  ACES 
have  a  lower  probability  of  promotion  and  are  thus  more  likely  to  leave  the  sample  through 
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attrition.  If  soldiers  who  left  the  sample  through  attrition  are  dropped  from  the  analysis,  then  the 
estimated  ACES  program  effect  could  be  biased  high.  Counting  the  soldiers  who  left  the  sample 
through  attrition  as  “not  promoted”  would  also  bias  the  findings. 

Censored  data.  The  problem  of  censored  data  is  a  general  problem  that  includes  sample 
attrition  as  a  special  case.  Censoring  occurs  when  an  event  of  interest  (e.g,,  participation  in  a  CE 
program,  promotion,  or  reenlistment)  cannot  be  observed,  either  because  it  occurs  outside  the 
period  over  which  the  data  are  obtained,  or  because  other  events  make  this  variable  impossible  to 
detect.  This  concept  is  relevant  to  the  evaluation  of  ACES  because  complete  data  on  ACES 
participation  is  unavailable  prior  to  1999.  Thus  an  evaluation  of  ACES  programs  would  be 
affected  by  “left”  censoring,  which  occurs  when  the  event  takes  place  prior  to  the  observation 
period.  “Right”  censoring  occurs  when  the  event  happens  after  the  observation  period.  Sample 
attrition  can  be  viewed  as  an  example  of  censoring  in  which  the  censoring  event  occurs  during 
the  observation  period. 

Measurement  Error.  Measurement  error  occurs  when  precise  measures  of  a  particular 
variable  of  interest  may  not  be  available.  This  may  occur  because  no  physical  measure 
corresponding  to  the  variable  of  interest  is  available  (e.g.,  intelligence  or  experience),  or  because 
the  variable  is  not  measured  consistently.  The  bias  introduced  by  measurement  error  can  be 
severe  (Green,  1997).  Four  sources  of  measurement  error  were  evident  in  the  studies  we 
reviewed. 

The  first  source  of  measurement  error  is  associated  with  CE  program  participation. 
Measurement  error  in  this  variable  can  occur  for  many  reasons — including  poor  records  of 
members’  CE  program  participation.  If  members  who  participated  in  a  CE  program  are  recorded 
as  non-participants,  either  because  of  poor  data  recording  or  censoring,  then  the  effect  is  to 
attenuate  (or  bias  towards  zero)  the  measured  ACES  effect  on  the  dependent  variable.  A  previous 
study  of  ACES  (Brink,  Newman,  Spurgeon,  &  Stock,  1981)  found  missing  ACES  participation 
data  to  be  a  common  phenomenon. 

The  second  source  of  measurement  error  is  associated  with  the  measure  of  retention. 
Studies  of  employee  turnover  in  the  civilian  literature  note  the  problem  of  measurement  bias  in 
measuring  turnover — both  when  using  survey  data  and  when  using  administrative  data  (Griffeth 
and  Horn,  1995).  At  issue  is  how  a  separation  is  categorized.  In  general,  evaluations  of  retention 
are  interested  in  determining  what  factors  can  decrease  voluntary  turnover.  Thus,  these  studies 
often  omit  involuntary  separations  (e.g.,  employees  who  are  fired  or  who  leave  for  death  or 
medical  reasons).  Some  survey  respondents  may  not  accurately  categorize  their  separation  as 
voluntary.  That  is,  they  may  give  more  socially  desirable  reasons  for  quitting  than  do  their 
employers.  Likewise,  reasons  for  separation  in  administrative  databases  may  not  be  completely 
accurate.  For  example,  a  person  who  separated  involuntarily  (e.g.,  fired)  may  be  categorized  as  a 
voluntary  separation  (e.g.,  laid  off)  to  make  the  separate  employee  eligible  for  unemployment 
compensation,  or  to  avoid  the  possibility  of  litigation.  The  military  literature  that  we  reviewed 
did  not  investigate  why  people  leave. 

Job  performance  measures  represent  a  third  source  of  measurement  error.  As  discussed 
previously,  measures  of  job  performance  are  not  readily  available  for  soldiers.  Furthermore, 
measures  of  job  performance  will  vary  by  the  type  of  work  soldiers  perform— which  can  differ 
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substantially  across  soldiers.  Consequently,  the  studies  we  reviewed  that  analyze  the  impact  of 
CE  participation  on  job  performance  use  promotions  (and  in  some  eases  demotions)  as  a  proxy 
for  performance.  Although  promotions  generally  are  indicators  of  good  performance,  there  are 
numerous  factors  other  than  performance  that  are  determinants  of  promotions.  Some  of  these 
variables  are  observable  and  can  be  controlled  for  in  a  regression  model  (e.g.,  time  in  grade, 
MOS).  Other  factors  are  less  observable  to  the  researcher  (e.g.,  number  of  promotion  positions 
available).  The  main  issue,  though,  is  that  because  promotions  are  an  imprecise  measure  of 
performance,  studies  of  the  impact  of  CE  participation  on  promotions  does  not  capture  the  “true” 
relationship  between  CE  participation  and  job  performance. 

Finally,  measures  of  cognitive  ability/intellect  represent  a  fourth  source  of  measurement 
error.  Soldiers’  ability  and  intelligence  are  important  determinants  in  the  propensity  to  participate 
in  a  CE  program  and  the  likelihood  of  promotion.  Researchers  have  used  different  variables  as 
proxies  for  ability  and  intelligence.  For  example,  the  most  common  measures  include  AFQT 
score  (e.g.,  Garcia  et  al.,  1998;  Boesel  &  Johnson,  1988),  high  school  diploma  (e.g.,  Garcia  et  al., 
1998),  early  promotion  or  special  qualification  by  a  review  board  (e.g.,  Fuchs,  1996;  Talaga, 
1994),  and  grade  point  average  (e.g.,  Fuchs,  1996;  Talaga,  1994).  These  variables  are  only 
proxies  for  ability  and  intelligence,  so  the  “true”  relationship  between  ability/intelligence  and  the 
dependent  variable  of  interest  is  unknown  and  the  estimated  relationship  is  biased  towards 
zero — or  no  effect  (Green,  1997).  Unfortunately,  a  poorly  measured  variable  can  bias  (in 
unknown  directions)  the  estimates  for  other  explanatory  variables  in  the  multivariate  regression 
model.  Although  including  a  variable  measured  with  error  in  the  regression  model  reduces  the 
reliability  of  the  estimated  relationship  between  CE  program  participation  and  the  outcome  of 
interest,  omitting  the  variable  could  cause  a  worse  problem. 

Sampling  Issues 

Two  sampling  issues  addressed  in  the  literature  that  are  relevant  to  this  study  are 
sampling  error  and  sample  sources. 

Sampling  Error .  Most  of  the  military  studies  that  we  reviewed  were  based  on  relatively 
large  samples.  For  example,  Boesel  and  Johnson’s  (1988)  study  of  DoD’s  Tuition  Assistance 
program  was  based  on  nearly  100,000  members  of  the  military.  However,  when  one  desires  to 
analyze  subsets  of  the  sample,  sampling  error  becomes  an  increasingly  important  issue.  For 
example,  when  Boesel  and  Johnson  analyzed  only  those  members  in  the  Navy  who  had 
participated  in  the  TA  program  and  who  responded  in  the  survey  that  they  were  “almost  sure”  or 
“certain”  of  promotion,  then  the  sample  size  dropped  to  approximately  84.  In  general,  larger 
samples  result  in  more  precise  estimates  of  the  impact  of  CE  participation  on  the  outcome  of 
interest.  That  is,  one  is  more  confident  of  findings  that  are  based  on  larger  samples  than  findings 
based  on  smaller  samples. 

Sample  sources.  The  primary  source  of  information  for  the  military  studies  we  reviewed 
was  administrative  databases.  In  general,  the  authors  of  the  studies  merged  administrative 
records  on  CE  program  participation  with  a  “master”  file  that  contained  information  on  soldiers’ 
career  history.  The  master  file  used  in  the  analysis  typically  contained  information  on  the 
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soldiers’  demographic  characteristics,  job  characteristics,  and  the  outcome  of  interest  (e.g., 
promotion  or  retention)  for  a  cohort  of  soldiers.12 

One  of  the  main  limitations  of  administrative  data  is  that  vital  information  on  soldiers’ 
unobservable  attributes  (e.g.,  intentions,  perceptions,  and  satisfaction)  is  not  available.  As  a 
result,  the  findings  of  various  studies  are  clouded  by  factors  that  the  researchers  cannot  control. 
Boesel  and  Johnson,  in  their  study  of  DoD’s  Tuition  Assistance  program,  had  the  unique 
opportunity  to  merge  administrative  records  with  the  1985  DoD  Survey.  This  allowed  the  authors 
to  compare  the  outcomes  of  interest  (i.e.,  reenlistment  and  promotion),  by  TA  participation 
status,  stratified  by  how  survey  respondents  answered  various  questions  in  the  survey.  They 
found,  for  example,  that  soldiers  who  had  never  participated  in  a  TA  program  were  intending  to 
leave  the  military  at  higher  rates  than  soldiers  who  had  previously  participated  in  a  TA  program. 
Their  findings  are  likely  biased,  however,  for  failing  to  control  for  factors  that  are  correlated  with 
both  participation  in  a  TA  program  and  intention  to  remain  in  the  military — such  as  time  in 
service. 


Summary  and  Implications 

The  research  literature  provides  limited  coverage  of  CE  programs,  focusing  primarily  on 
tuition  assistance  and  basic  skills  programs.  Within  this  limited  range,  the  research  gives  a 
relatively  positive  picture  of  the  effects  of  participation  in  these  programs  on  retention  and 
performance.  This  section  summarizes  the  research  findings,  describes  some  of  the  limitations  of 
these  findings,  and  makes  recommendations  for  the  evaluation  of  ACES  based  on  these  results. 

Summary  of  Findings 

The  literature  provides  a  fairly  consistent  picture  of  both  the  motivations  to  take  part  in 
CE  and  effects  of  CE  on  recruitment,  retention,  and  performance.  Despite  differences  between 
military  and  civilian  employment  environments,  the  conclusions  of  research  in  these  two  areas 
are  consistent. 

Continuing  education  serves  both  organizational  and  personal  goals.  A  program  such  as 
ACES  provides  an  opportunity  for  a  soldier  to  improve  performance  on  his  or  her  military 
mission  and  to  better  prepare  for  later  civilian  employment.  Given  the  divergent  goals  that  may 
be  served  by  CE,  it  is  not  surprising  that  the  motivations  for  participation  are  complex  and 
include  both  internal  and  external  factors.  Despite  the  divergence  in  motivations,  it  seems  to  be  a 
fair  characterization  of  the  situation  that  those  who  participate  in  CE  tend  to  be  better  qualified 
and  more  highly  motivated  soldiers  (or  employees)  than  those  who  don’t.  This  difference 
confirms  our  concern  that  the  evaluation  plan  must  control  for  selection  bias. 

Although  there  is  little  direct  evidence  that  opportunities  for  CE  enhance  recruitment, 
circumstantial  evidence  would  suggest  that  they  do.  Overall,  educational  benefits  are  a  principal 
reason  for  enlisting.  Although  the  G.I.  Bill  and  the  ACF  are  the  most  well  publicized  educational 


12  A  cohort  is  typically  defined  by  when  soldiers  entered  the  military  or  were  eligible  for  a  specific  event  (e.g., 
reenlistment  or  promotion),  or  by  military  rank. 
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benefits,  CE  seems  likely  to  be  a  contributing  factor.  The  importance  of  educational  benefits  in 
private  industry  would  also  suggest  that  they  have  a  positive  effect  on  recruitment,  since  private 
industry  has  no  program  that  is  analogous  to  veterans’  benefits. 

A  positive  effect  of  CE  on  performance  is  reflected  in  the  opinions  of  officers  and 
enlisted  personnel,  promotion  rates,  and  actual  performance  ratings.  Servicemembers  indicated 
that  they  believed  that  CE  would  improve  job  performance,  particularly  for  enlisted  personnel. 
Although  there  is  little  data  to  judge  whether  these  perceptions  are  accurate,  the  effect  of 
participation  in  CE  on  promotion  was  positive,  even  after  controlling  for  the  effects  of 
moderating  variables.  In  correspondence  with  servicemember  opinions,  the  effects  on  promotion 
rates  were  stronger  for  enlisted  personnel. 

Turnover  covers  both  retention  and  attrition.  Regarding  retention,  existing  research  paints 
a  clear  picture  in  which  consistent  participation  in  CE  (particularly  tuition  assistance)  increases 
the  likelihood  that  servicemembers  will  reenlist.  The  effect  remains,  albeit  at  a  reduced 
magnitude,  when  effects  of  other  factors  are  controlled  statistically.  There  are  some  exceptions  to 
the  general  finding,  such  as  the  Brauchle’s  (1998)  result  indicating  that  servicemembers  who 
intend  to  separate  from  military  service  also  participate  in  tuition  assistance,  perhaps  to  prepare 
for  their  civilian  career. 

We  found  limited  evidence  that  basic  skills  education  may  reduce  attrition  substantially, 
but  no  research  that  examines  the  effect  of  other  CE  programs  on  attrition.  Because  basic  skills 
education  can  occur  early  in  a  soldier’s  career,  it  has  the  potential  to  affect  attrition,  most  of 
which  also  occurs  early.  We  suspect  that  other  programs,  such  as  tuition  assistance  would  not 
have  a  substantial  effect  on  attrition,  which  tends  to  occur  before  the  soldier  has  had  much 
opportunity  to  use  tuition  assistance. 

Limitations  of  Results 

Several  factors  limit  the  generality  of  the  findings  of  past  research.  First,  with  few 
exceptions,  the  existing  research  evaluates  voluntary,  postsecondary  education  programs,  most 
notably  TA.  Although  some  of  the  relationships  that  were  found  for  TA  programs  were 
confirmed  for  the  BSEP  program,  no  research  was  found  addressing  other  ACES  programs,  or 
their  counterparts  in  either  the  other  Military  Services  or  the  civilian  workplace. 

Taken  as  a  whole,  the  results  reaffirm  the  importance  of  considering  selection  bias  in 
evaluating  the  effects  of  CE  programs.  Available  evidence  indicates  that  the  individuals  who 
participate  in  CE  programs  tend  to  be  better  qualified  and  more  highly  motivated  than  those  who 
don’t.  Consequently,  effects  of  CE  participation  on  retention  or  performance  are  reduced  when 
attempts  are  made  to  control  for  selection  bias.  Results  of  studies  in  which  selection  bias  was  not 
controlled  for  should  be  viewed  with  caution.  The  simplest  presentation  of  results,  in  this  case, 
may  be  misleading. 

One  limitation  of  several  studies  reviewed  is  that  the  authors  combined  data  on 
servicemembers  in  different  stages  of  their  military  career  instead  of  estimating  different  models 
for  different  types  of  members.  Assuming  that  the  relationship  between  CE  participation  and  the 
outcome  of  interest  is  fixed  across  all  types  of  service  members  could  bias  the  findings.  For 
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example,  the  motivation  for  participating  in  a  CE  program  may  be  much  different  for  a  soldier  in 
his  or  her  first  enlistment  term  than  in  his  second  enlistment  term.  Consequently,  the  relationship 
between  the  dependent  variable  (e.g.,  retention)  and  the  explanatory  variables  (including  CE 
participation)  may  be  different  for  the  different  types  of  members.  An  analysis  of  the  retention 
effects  of  CE  participation  for  members  in  their  first  enlistment  could  have  significantly  different 
findings  than  an  analysis  of  the  retention  effects  of  CE  participation  for  members  in  their  second 
enlistment. 

The  existing  studies  were  not  guided  by  an  overall  conceptual  modeling  framework.  Such 
a  framework  can  guide  the  selection  and  operational  definition  of  outcome  and  control  variables. 

Recommendations  for  the  Evaluation 

For  the  most  part,  the  outcome  measures  addressed  in  previous  research,  retention  and 
promotions,  are  reasonable  to  include  in  an  evaluation  of  most  ACES  programs.  They  are 
important  to  the  Army,  and  previous  research  has  shown  that  educational  benefits  can  affect 
these  outcomes.  To  the  extent  that  other  performance  measures  can  be  identified  in  existing 
personnel  databases,  they  should  be  addressed  as  well.  Improvements  in  these  more  direct 
measures  of  performance  would  not  be  confounded  by  interactions  with  other  variables,  such  as 
time  in  service.  Evaluation  of  the  effects  of  CE  on  attrition  should  focus  on  programs,  such  as 
FAST,  that  occur  early  in  a  soldier’s  career. 

The  need  to  control  for  selection  bias  has  been  stressed  several  times  in  this  review.  We 
believe  that  the  process  of  controlling  for  selection  bias  would  be  aided  by  using  a  general  model 
of  the  retention  or  promotion  process  to  guide  the  data  analysis,  including  the  identification  and 
selection  of  control  variables.  In  other  words,  the  analysis  needs  to  consider  more  than  simply 
whether  a  soldier  participated  in  ACES  and  a  single  control  variable  (e.g.,  the  Inverse  Mills 
Ratio).  A  more  general  model  of  retention  or  promotion  is  required,  that  includes  additional 
explanatory  variables  not  necessarily  related  to  ACES  participation.  The  use  of  such  a  model 
would  also  aid  the  interpretation  of  results,  as  well  as  in  forecasting  the  results  of  policy  changes. 

Several  other  sources  of  bias  should  be  considered  in  the  evaluation  design  and  analysis 
plan.  Limits  of  the  data — sample  attrition,  censoring,  measurement  error,  and  sampling  error — 
should  be  considered  for  the  variables  selected  for  inclusion  in  the  evaluation.  The  sample  size 
and  analytical  methods  should  be  devised  to  minimize  the  effects  of  these  errors. 


32 


EVALUATION  PLAN 


Because  ACES  participation  is  voluntary  and  available  to  all  soldiers,  it  is  not  possible  to 
design  a  controlled  experiment  in  which  a  randomly  determined  subset  of  soldiers  are  eligible  to 
participate  in  the  program.  One  major  implication  of  evaluating  a  program  without  random 
assignment  to  a  test  and  control  group  is  that  soldiers  who  participate  in  the  program  could  be 
systematically  different  from  soldiers  who  do  not  participate.  Furthermore,  many  of  the  factors 
that  increase  the  propensity  of  soldiers  to  participate  in  ACES  (e.g.,  motivation  and  ability)  are 
likely  to  affect  the  outcomes  of  interest  (i.e.,  retention  and  job  performance).  Thus,  the  evaluator 
must  identify  which  differences  in  outcomes  between  the  test  and  control  groups  should  be 
attributed  to  ACES,  and  which  differences  should  be  attributed  to  underlying  differences 
between  participants  and  non-participants. 

This  evaluation  plan  draws  from  the  results  reported  in  the  literature  reviewed  in  the 
previous  section.  It  also  considers  constraints  brought  about  by  the  availability,  accuracy,  and 
completeness  of  data  indicating  program  participation  and  critical  outcome  variables  reflecting 
soldier  retention  and  performance.  The  literature  review  identified  explanatory  variables  that 
should  be  collected  for  the  evaluation  and  recommended  methodological  approaches  and 
statistical  techniques  that  should  be  used  to  conduct  the  evaluation.  Following  this  section,  the 
database  development  plan  provides  a  detailed  discussion  of  what  variables  should  be  included 
in  the  evaluation  database  and  how  that  database  should  be  constructed.  The  database 
development  plan  and  the  evaluation  plan  are  closely  linked.  A  major  purpose  of  the  evaluation 
plan  is  to  help  guide  the  data  collection  process.  Similarly,  data  availability,  as  documented  in 
the  database  development  plan,  affects  how  the  evaluation  can  be  conducted. 

Purpose  of  the  Evaluation  Plan 

Informed  policy  decisions  require  a  comprehensive  and  technically  sound  evaluation  of 
the  ACES  program  and  the  benefits  it  provides  to  the  Army  and  its  members.  The  main  purpose 
of  this  evaluation  plan  is  to  assure  a  successful  evaluation  by  identifying  (a)  questions  that  the 
evaluation  should  answer,  (b)  data  that  should  be  collected  to  answer  these  questions,  (c)  a 
technical  approach  to  analyze  the  data,  and  (d)  statistical  and  data  issues  that  the  evaluation  must 
address. 

This  evaluation  plan  is  built  on  a  solid  theoretical  foundation  and  several  decades  of 
applied  research  on  the  topics  of  Army  manpower  planning,  program  evaluation,  and  the  benefits 
of  education  and  training.  The  methods  and  models  proposed  in  this  plan  build  on  the  applied 
research  conducted  during  the  past  two  decades  in  the  areas  of  staff  retention,  performance,  and 
evaluation  of  education  and  training  programs. 

In  summary,  an  evaluation  plan  helps  assure  a  systematic  evaluation  of  the  data  using 
structurally  sound  and  complete  models  and  appropriate  modeling  techniques.  A  thorough 
evaluation  increases  the  likelihood  that  the  study  will  provide  policy-relevant  and  scientifically 
sound  information  regarding  the  contribution  of  ACES  to  Army  readiness. 
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Evaluation  Questions 


The  central  issue  for  this  evaluation  is  to  determine  the  impact  of  the  ACES  program  on 
combat  readiness.13  The  hypothesis  is  that  ACES  improves  combat  readiness  indirectly,  through 
effects  on  recruiting,  retention,  and  performance.  The  U.S.  Total  Army  Personnel  Command 
(PERSCOM)  has  requested  that  the  evaluation  focus  on  the  two  latter  issues.  The  evaluation 
plan,  therefore,  is  designed  around  the  following  questions: 

1 .  What  effect  does  ACES  have  on  soldier  retention,  as  reflected  by  such  outcomes  as  early 
attrition  and  likelihood  of  reenlistment? 

2.  What  effect  does  ACES  have  on  soldier  performance,  as  indicated  by  the  likelihood  and 
timing  of  promotions  and  Military  Occupational  Specialty  (MOS)  reclassification  actions? 

3.  What  is  the  net  benefit,  or  value  added,  of  ACES  to  the  Total  Army? 

These  three  questions  generate  numerous  other  questions  regarding  the  characteristics  of  soldiers 
who  participate  in  the  ACES  program,  whether  the  program  benefits  vary  by  type  of  soldier,  the 
most  probable  timing  of  soldier  participation  in  ACES,  and  whether  certain  ACES  programs  are 
more  cost  effective  in  terms  of  their  contribution  to  combat  readiness.  Below  is  a  more  detailed 
list  of  questions  that  the  evaluation  should  attempt  to  answer. 

Soldier  retention  analysis. 

1 .  Does  the  ACES  program  increase  the  propensity  of  soldiers  to  reenlist?  If  so, 

•  Does  ACES  participation  increase  reenlistments  at  the  end  of  the  first  term,  second  term, 
and  additional  terms? 

•  Which  ACES  programs  increase  or  decrease  the  propensity  of  soldiers  to  reenlist? 

•  By  how  much  does  participation  in  specific  ACES  programs  increase  or  decrease  the 
propensity  of  soldiers  to  reenlist? 

•  Does  the  ACES  reenlistment  effect  differ  by  type  of  soldier  (e.g.,  high  school  graduates 

versus  non-graduates,  soldiers  with  different  career  intentions)  or  by  job  attributes  (e.g 
by  MOS)?  '  '  ' 

2.  Does  ACES  decrease  attrition?  If  so, 

•  By  how  much  does  ACES  decrease  early  attrition? 

•  Which  ACES  programs  decrease  attrition? 


3  This  evaluation  focuses  on  the  costs  and  benefits  of  ACES  to  the  Army,  although  participation  in  continuing 
education  also  has  important  implications  for  the  soldier,  for  his  or  her  family,  and  for  society. 
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•  Does  any  decrease  in  attrition  differ  by  type  of  soldier  (e.g.,  high  school  graduates  versus 
non-graduates,  soldiers  with  different  career  intentions)  or  by  job  attributes  (e.g.,  by 
MOS)? 

Soldier  performance  analysis. 

3.  Does  participation  in  ACES  programs  increase  job  performance  as  measured  by  the 
occurrence  and  timing  of  promotions,  and  by  reclassification  actions?  If  so, 

•  Does  ACES  participation  increase  the  likelihood  that  a  soldier  will  be  promoted? 

•  Does  ACES  participation  reduce  the  expected  time  to  promotion? 

•  Which  ACES  programs  are  best  able  to  increase  the  likelihood  or  decrease  the  expected 
time  to  promotion? 

•  Does  ACES  participation  increase  the  likelihood  that  a  soldier  will  be  reclassified  to 
another  MOS  to  further  his  or  her  career? 

•  Which  ACES  programs  are  most  likely  to  lead  to  MOS  reclassification? 

•  Does  the  impact  of  ACES  participation  on  promotions  differ  by  type  of  soldier  (e.g.,  high 
school  graduates  versus  non-graduates)  or  by  rank? 

Cost-benefit  analysis. 

4.  What  is  the  net  value  of  ACES  to  the  Army  in  terms  of  a  cost-benefit  analysis? 

•  What  is  the  net  present  value  of  various  ACES  programs? 

•  Which  ACES  programs  are  most  valued  by  the  members? 

•  How  does  the  cost  per  reenlistment  attributed  to  the  ACES  program  compare  to  the  cost 
per  reenlistment  attributed  to  other  programs  (e.g.,  selected  reenlistment  bonuses)? 

Outline  of  the  Evaluation  Plan 

The  remainder  of  this  plan  discusses  the  data  and  methods  proposed  to  answer  the 
questions  posed  above.  The  next  section  covers  the  scope  of  the  evaluation.  It  considers  which 
ACES  programs  to  evaluate,  the  time  period  that  will  be  covered  in  the  evaluation,  and  the 
outcomes  that  will  be  explicitly  defined  and  evaluated.  Following  that  section  is  an  overview  of 
the  most  appropriate  modeling  techniques  to  conduct  the  evaluation.  Then  the  evaluation  plans 
for  the  retention  and  job  performance  analyses  are  presented  in  two  sections.  Each  of  these 
sections  addresses  the  following  three  concepts: 

1 .  The  development  of  a  conceptual  model  to  identify  particular  hypotheses  to  be  tested 
concerning  the  contribution  of  ACES  to  Army  readiness; 


35 


2.  The  identification  of  an  appropriate  functional  form,  statistical  techniques,  and  data  to 
estimate  the  contribution  of  ACES  to  Army  readiness;  and 

3.  The  identification  of  appropriate  statistical  tests  to  evaluate  the  model  and  its  components 
and  to  test  hypotheses  regarding  the  effect  of  ACES  on  measures  of  Army  readiness. 

Following  these  two  plans  is  a  discussion  of  the  data  and  methods  to  conduct  a  cost-benefit 
analysis  of  ACES,  Approaches  to  quantify  the  benefits  and  costs,  in  dollars,  are  discussed.  The 
final  section  contains  a  brief  summary  of  the  plan. 

Scope  of  the  Evaluation 

ACES  provides  a  wide  range  of  programs  and  services  to  support  the  needs  of  the  Army 
and  to  support  the  professional  and  personal  development  of  soldiers  in  the  area  of  education. 
ACES  programs  vary  in  terms  of  their  resources,  number  of  participants,  and  perceived 
importance  of  their  contribution  to  Army  readiness.  Likewise,  different  ACES  programs  and 
services  are  designed  to  benefit  soldiers  at  different  stages  in  their  military  career. 

This  section  contains  a  brief  overview  of  ACES  programs  and  discusses  four  criteria  used 
to  select  those  programs  for  which  empirical  evaluation  is  most  relevant,  viable,  and  cost 
effective.  In  addition,  measures  of  ACES  participation,  measures  of  the  contribution  of  ACES  to 
Army  readiness,  and  the  time  period  over  which  to  evaluate  ACES  are  discussed. 

ACES  Programs  and  Criteria  Used  to  Select  Programs  for  Evaluation 

ACES  comprises  several  programs  and  services.  Five  of  these  programs  meet  the  criteria 
to  be  included  in  an  empirical  evaluation:  (a)  the  Army  Tuition  Assistance  (TA)  Program,  (b)  the 
Servieemembers  Opportunity  Colleges  Army  Degree  (SOCAD)  Program,  (c)  the  Functional 
Academic  Skills  Training  (FAST)  program,  (d)  the  MOS  Improvement  Courses,  and  (e)  Non¬ 
commissioned  Officer  (NCO)  Leader  Development  Courses.14  The  following  criteria  were  used 
to  select  these  five  programs  for  evaluation.1 '‘‘Many  of  these  criteria  are  inter-related. 


14  These  programs  and  services  are  similar  to  components  of  the  Navy’s  VOTED  Program  included  in  a  recent 
evaluation  conducted  by  the  Center  For  Naval  Analysis  (Garcia,  Joy  and  Reese,  1998).  Components  of  VOLED 
evaluated  by  CNA  study  include:  (a)  tuition  assistance  (with  the  same  funding  criteria  as  ACES),  (b)  the  Program 
for  Afloat  College  Education  (PACE),  (c)  Academic  Skills  Learning  Centers,  and  (d)  Counseling. 

15  An  evaluation  of  ACES  conducted  two  decades  ago  (Brink  et  al„  1981)  was  designed  to  determine  the  effects  of 
ACES  program  participation  on  soldier  performance.  That  study  proposed  to  evaluate  four  ACES  programs  that 
existed  at  that  time:  (a)  Basic  Skills  Education  Program  I,  Literacy  Phase  (BSEP  I-Lit);  (b)  Basic  Skills  Program  I, 
English  as  a  Second  Language  (BSEP  I-ESL);  (c)  Skill  Development:  General  Vocational-Technical  (Vo-Tech);  and 
(d)  Veterans  Educational  Assistance  Program  (VEAP),  These  four  components  were  selected  by  the  research  team 
and  the  government’s  contract  representatives  based  on  eight  criteria:  (a)  number  of  participants,  (b)  adequate  size  of 
participant  and  non-participant  groups,  (c)  available  and  accurate  participant  data,  (d)  operational  indicator  of 
program  completion  or  degree  of  participation  in  the  program,  (e)  probable  impact  on  military  proficiency,  (f) 
probable  impact  on  Army  career  progression,  (g)  ease  of  data  collection,  and  (h)  perceived  significance  to  the  Army. 
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•  Level  of  funding.  Level  of  funding  is  an  important  criterion  for  two  reasons.  First, 
programs  with  more  funding  would  be  expected  to  have  a  larger  impact  on  Army 
readiness  than  programs  with  less  funding,  so  the  impact  is  likely  to  be  easier  to  detect. 
Second,  to  some  degree,  programs  with  more  funding  are  likely  to  be  more  vital  to  the 
Army’s  mission. 

•  Number  of  participants.  Programs  with  a  large  number  of  participants  were  more  likely 
to  be  selected  for  evaluation  than  programs  with  fewer  participants  for  several  reasons. 
First,  the  accuracy  of  the  estimated  program  effect  increases  with  the  sample  size  (or 
number  of  participants).  Second,  programs  with  a  large  number  of  participants  are  more 
likely  to  have  a  detectable  impact  on  Army  readiness. 

•  Evaluability  of  the  program.  For  the  evaluation  to  be  effective,  there  must  be  (a)  an 
adequate  sample  size  for  both  the  participant  and  non-participant  groups,  (b)  a  logical 
link  between  program  participation  and  the  soldier  outcomes  of  interest,  and  (c)  sufficient 
impact  for  the  effect  to  be  detectable. 

•  Data  availability.  Reliable  data  on  program  participation  is  required  to  evaluate  a 
program.  The  three  main  sources  of  program  participation  data  are  the  Education 
Management  Information  System  (EDMIS),  the  Army/American  Council  on  Education 
Registry  Transcript  System  (AARTS),  and  databases  of  SOCAD  agreements  for  two-  and 
four-year  degrees.  In  addition,  the  1999  Survey  of  Active  Duty  Personnel  (SADP) 
includes  self-reported  participation  in  several  continuing  education  (CE)  programs.  These 
data  sources  are  described  in  greater  detail  in  the  database  development  plan. 

A  brief  description  of  1 1  ACES  programs  and  a  comparison  of  these  programs  using 
three  of  the  four  criteria  described  above  is  provided  in  Table  4  (level  of  funding  was  excluded 
from  the  table).  In  addition,  the  table  identifies  those  ACES  programs  that  are  “operational”  in 
nature.  Operational  programs  are  those  designed  to  benefit  the  military  directly,  although  the 
member  might  benefit  indirectly,  while  non-operational  programs  mainly  benefit  the  member  but 
might  indirectly  benefit  the  military.16  Participation  and  cost  estimates  for  all  programs  except 
SOCAD  are  taken  from  the  ACES  Quarterly  Participation,  Cost,  and  Evaluation  Report  for 
Fiscal  Year  (FY)  1999. 


For  example,  MOS  improvement  courses  offered  through  ACES  are  designed  primarily  to  improve  the  member’s 
job  performance.  For  many  college  courses  funded  in  part  through  Tuition  Assistance,  however,  the  member  is 
likely  the  primary  beneficiary  with  the  Army  benefiting  indirectly. 
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Table  4 

Application  of  Criteria  to  Determine  Which  ACES  Components  to  Evaluate 
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Table  4 

Application  of  Criteria  to  Determine  Which  ACES  Components  to  Evaluate 
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Table  4 


40 


Centers  installations  to  provide  soldiers  with  access  to  multi-  by  soldiers  assess  benefits  EDMIS,  but 

media  computers,  the  Internet,  cable  and  video  for  this  support  may  not  be 

teleconferencing,  classrooms,  and  testing  facilities  to  program.  maintained  at 

facilitate  the  continuing  education  process.  all  sites. 


Programs  were  excluded  from  the  evaluation  for  one  or  more  of  three  reasons:  (a) 
insufficient  number  of  participants  or  non-participants,  (b)  unavailability  or  inaccuracy  of 
participation  data,  and  (c)  impact  on  retention  or  performance  expected  to  be  limited  in  scope. 
For  two  of  the  programs,  the  High  School  Completion  and  the  English  as  a  Second  Language 
(ESL)  Programs,  the  participation  rate  was  considered  to  be  too  low  to  obtain  a  reasonably  sized 
sample  of  participants  for  evaluation.  Counseling,  on  the  other  hand,  had  a  very  large  number  of 
participants,  which  would  lead  to  a  small  non-participant  population.  In  addition,  the  information 
we  received  from  PERSCOM  indicated  that  available  participation  data  in  EDMIS  do  not 
reliably  record  whether  counseling  was  conducted  for  educational  purposes  or  for  some  other 
reason.  Similar  potential  data  problems  were  brought  up  for  Army  Education  Centers. 
Participation  in  the  ACES  Testing  Program,  which  consists  of  Civilian/Academic  Testing  and 
Army  Personnel  Testing,  is  recorded  in  AARTS,  but  only  for  tests  that  are  passed.  This  program 
was  excluded  because  the  participation  data  would  give  a  biased  view  of  the  benefits  of  the 
program.  The  Foreign  Language  Training  Program  primarily  serves  the  immediate,  mission- 
related  needs  of  soldiers  who  are  stationed  overseas,  and  was  not  expected  to  have  any 
substantial  long-term  effect  on  retention  or  performance. 

Outcome  Measures  Evaluated 

The  purpose  of  this  evaluation,  as  discussed  above,  is  to  determine  the  impact  of  ACES 
on  Army  readiness.  This  evaluation  focuses  on  two  ways  that  ACES  participation  is 
hypothesized  to  affect  Army  readiness:  (a)  by  improving  staffing  through  lower  early  attrition 
and  higher  retention,  and  (b)  by  improving  job  performance  (as  measured  by  promotion  and 
reclassification).  As  noted  previously,  “operational”  programs  such  as  the  MOS  Improvement 
Courses  and  the  NCO  Leader  Development  Courses  are  designed  primarily  to  improve  job 
performance,  while  non-operational  programs  such  as  Testing,  FAST,  Tuition  Assistance,  and 
Counseling  provide  soldiers  with  more  general  skills  that  indirectly  benefit  the  Army  (see  Figure 
1).  Also,  soldiers  who  perform  better  in  their  job  are  more  likely  to  be  promoted,  and  higher  pay 
resulting  from  the  promotion  might  increase  the  probability  of  reenlistment. 

Retention.  Two  measures  of  soldier  retention  can  be  evaluated:  (a)  reenlistments  at  the 
first,  second,  or  greater  reenlistment  decision;  and  (b)  early  first-term  attrition.  Because  the  vast 
majority  of  attrition  occurs  in  the  first  term,  there  is  little  to  be  gained  from  looking  at  attrition  in 
later  enlistment  terms.  Reenlistment  can  be  modeled  as  a  dichotomous  outcome:  reenlisted 
versus  did  not  reenlist.17  To  complete  this  analysis,  one  must  identify  several  groups  of  soldiers: 
(a)  soldiers  who  reached  a  reenlistment  decision  point,  (b)  soldiers  at  a  reenlistment  point  who 
chose  to  reenlist,  and  (c)  soldiers  who  chose  not  to  reenlist.  In  addition,  to  analyze  early  attrition, 
one  must  identify  which  soldiers  separated  during  the  period  for  which  data  are  collected  and  the 
reason  for  separation.  (Appendix  A  contains  a  list  of  separation  codes  used  to  distinguish 
attrition  from  the  decision  not  to  reenlist).  These  groups  are  described  in  the  following 
discussion. 


17  Soldiers  may  also  extend  their  current  enlistment  contract.  Although  it  seems  clear  that  extensions  of  a  few 
months  should  not  be  included  with  reenlistments,  it  may  be  appropriate  to  consider  longer  extensions,  such  as  those 
of  one  year  or  more,  as  being  equivalent  to  reenlistments. 
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Participates  in: 

•MOS  Improvement 
Courses 
•NCO  Leader 
Development  Courses 


Soldier 


\ 


- r- 

Participates  in: 

i  *Soldiers’  satisfaction  with  the 

•Tuition  Assistance 

_ . 

Army 

•SOCAD 

•Soldiers’  job  satisfaction 

•FAST 

. . r 

•Soldiers’  civilian  earnings 

potential 

•Reclassification  for  new  MOS 

Figure  1.  Model  of  program  benefits. 


•  Soldiers  at  a  reenlistment  decision  point.  This  group  is  defined  as  those  soldiers  who 
did  not  separate  prior  to  completing  their  enlistment  contract  and  who  were  not  ineligible 
to  reenlist  because  of  medical  or  psychological  reasons.  Soldiers  ineligible  to  reenlist  for 
disciplinary  reasons  or  for  not  meeting  other  requirements  within  the  soldiers’  control 
(e.g.,  failure  to  obtain  a  GED  for  soldiers  enlisted  without  a  high  school  diploma  or  GED) 
are  included  in  this  group. 

•  Soldier  who  chose  to  reenlist.  Soldiers  who  reenlist  at,  or  prior  to,  the  expiration  of  term 
of  service  (ETS)  date  of  their  current  enlistment  contract  are  considered  reenlistments. 

•  Soldiers  who  “voluntarily”  separate  during  the  data  collection  period.  This  category 
includes  soldiers  who  separate  for  non-medical  reasons  during  the  data  collection  period. 
It  is  important  to  distinguish  between  “voluntary”  attrition,  defined  as  separations  that  are 
the  result  of  the  soldier’s  actions  (e.g.,  the  decision  to  leave,  poor  performance  or 
unacceptable  behavior),  from  “involuntary”  attrition,  separations  for  medical  or 
psychological  reasons.  As  discussed  previously,  there  are  some  concerns  regarding  the 
ability  of  the  Army’s  separation  data  to  separate  attrition  into  meaningful  categories.  To 
the  extent  possible,  though,  involuntary  separations  that  can  be  identified  should  be 
excluded  from  the  attrition  analysis. 

Performance.  Little  information  on  job  performance  is  systematically  collected  and 
stored  by  the  Army.  Measures  that  are  collected  typically  are  not  comprehensive,  but  deal  with 
specific  requirements  (e.g.,  marksmanship,  level  of  physical  fitness)  or  pertain  to  specific 
military  occupational  specialties  (e.g.,  scores  on  proficiency  tests).  For  this  reason,  researchers 
often  use  promotions,  and  especially  early  promotions,  as  an  indicator  of  superior  performance. 
Promotions,  though,  are  imprecise  measures  of  job  performance  because  factors  other  than 
performance,  such  as  time  requirements  and  the  availability  of  promotion  slots,  affect 
promotions. 

The  evaluation  should  focus  on  promotions  to  E-5  and  E-6  for  several  reasons.  Because 
promotions  through  E-4  are  largely  automatic  based  on  time  in  service,  and  because  relatively 
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few  soldiers  participate  in  most  ACES  programs  (with  the  exception  of  FAST)  in  the  early  stages 
of  their  military  career  (e.g.,  as  E-ls  and  E-2s),  there  is  a  reduced  chance  of  finding  an  effect  of 
ACES  participation  on  promotions  to  grades  lower  than  E-5.  On  the  other  hand,  there  are  fewer 
NCOs  at  higher  grades  (E-7  and  above)  than  at  the  more  junior  levels,  reducing  the  sample  size. 
Furthermore,  promotions  at  the  higher  levels  depend  on  the  number  of  available  positions, 
which,  in  turn,  varies  significantly  with  career  field  and  other  variables  that  are  not  related  to 
performance.  Consequently,  these  higher  grades  also  have  lower  prospects  of  exhibiting  positive 
effects  of  ACES  participation. 

Reclassification  to  another  MOS  can  also  be  an  indicator  of  performance.  Some  MOS 
changes  may  be  coincident  with  promotions,  but  others  may  occur  if  a  soldier  becomes  qualified 
for  a  more  desirable  MOS  or  an  MOS  with  better  promotion  prospects.  Past  research  has  not 
examined  whether  participation  in  CE  increases  the  likelihood  of  reclassifications,  but  as  the 
conceptual  model  in  Figure  1  indicates,  there  is  some  reason  to  anticipate  that  participation  in 
some  programs  might  have  a  positive  effect  on  the  likelihood  of  MOS  reclassification. 

Sources  of  outcome  data.  A  primary  source  of  administrative  data  on  retention  and 
performance  that  will  be  used  for  the  evaluation  is  the  Active  Duty  Military  Personnel  Edit  File 
(PEF).  The  Defense  Enrollment  Eligibility  Reporting  System  (DEERS)  database  contains 
additional  relevant  administrative  information  regarding  the  benefits  for  which  soldiers  are 
eligible  (such  as  the  Montgomery  GI  Bill  [MGIB]).  In  addition,  ARI  has  developed  an  extensive 
database  on  the  FY  1999  accession  cohort,  called  Project  First-Term,  which  we  propose  to  use 
for  the  attrition  analysis.  Finally,  the  1999  SADP  asks  respondents  questions  regarding  their 
intentions  to  remain  in  the  military,  as  well  as  other  satisfaction  questions.  A  detailed  description 
of  the  databases  used  and  the  rationale  for  their  selection  is  included  in  the  database  development 
plan  presented  in  the  following  section. 

Period  of  Analysis  and  Cohorts 

Data  availability  and  the  stages  in  a  soldier’s  military  career  when  the  soldier  might 
participate  in  ACES  both  affect  the  period  and  the  cohorts  chosen  for  the  analysis.  Data  from 
EDMIS  are  available  for  all  installations  from  1999  forward,  and  from  a  subset  of  continental 
United  States  (CONUS)  installations  from  1995  forward.  Ideally,  historical  data  on  soldiers’ 
participation  in  ACES  from  accession  to  separation  would  be  available  for  the  evaluation.  The 
goal  for  completion  of  the  evaluation,  however,  limits  the  time  frame  for  collecting  data  through 
approximately  the  end  of  the  2001  calendar  year  (CY).  Consequently,  for  a  subset  of  the  Army 
there  could  potentially  be  seven  years’  data  on  ACES  participation,  while  for  the  entire  Army 
there  would  at  most  be  three  years’  data  on  ACES  participation  (i.e.,  CY  1999  through  CY 
2001).18 


To  increase  the  time  period  over  which  ACES  participation  data  are  available,  a  cohort  that  consists  of  the  subset 
of  soldiers  assigned  to  a  CONUS  installation  that  used  EDMIS  prior  to  1998  was  considered.  For  example,  the 
evaluation  could  analyze  a  cohort  that  consists  of  soldiers  who  (a)  were  stationed  to  one  of  the  30  CONUS 
installations  using  EDMIS  in  1996,  and  (b)  were  subsequently  never  assigned  to  an  installation  without  EDMIS. 
One  serious  problem  with  such  a  cohort  is  that  the  sample  would  be  non-random.  It  would  over-represent  soldiers 
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Soldiers  might  participate  in  different  ACES  programs  at  different  stages  in  their  military 
career.  For  example,  participants  in  FAST  tend  to  be  relatively  recent  accessions,  while 
participants  in  NCO  leadership  programs  are  NCOs  (or  soldiers  who  will  soon  be  promoted  to 
NCO),  Consequently,  participation  in  programs  like  Tuition  Assistance  and  NCO  leadership 
enhancement  will  likely  be  low  during  first  enlistment,  while  participation  in  FAST  will  likely  be 
much  higher  during  the  first  enlistment  than  in  subsequent  enlistments.19 

Given  these  data  constraints  and  the  nature  of  when  soldiers  typically  participate  in 
ACES,  the  evaluation  focuses  on  the  following  five  basic  cohorts. 

•  Cohort  1:  Non-prior  Service  Accessions  in  FY  1999.  This  cohort  consists  of  all  active 
duty,  enlisted,  non-prior  service  accessions  in  1999.  This  cohort  will  be  the  focus  of  the 
analysis  of  early  first-term  attrition;  it  will  also  be  used  to  examine  effects  of  ACES 
participation  on  early  reclassification.  The  primary  ACES  programs  that  will  be 
examined  in  this  cohort  are  FAST  and  MOS  Improvement  Courses,  because  these  two 
programs  are  expected  to  be  used  by  soldiers  early  in  their  first  term  of  enlistment.  The 
database  used  to  define  this  cohort  will  come  primarily  from  the  ARI  Project  First  Term 
attrition  database  and  EDMIS. 

•  Cohort  2a:  Soldiers  with  an  ETS  date  in  CY  2001.  This  cohort  consists  of  all  active 
duty,  enlisted  members  with  an  ETS  date  in  2001.  This  cohort  includes  soldiers  in  their 
first,  second,  or  greater  term  of  enlistment.  Although  a  complete  history  of  ACES 
participation  will  not  be  available  for  some  soldiers,  a  complete  history  of  ACES 
participation  in  the  two  to  three  years  prior  to  the  ETS  date  would  be  available.  The  main 
purpose  of  analyzing  this  cohort  is  to  determine  the  impact  of  ACES  on  retention — 
especially  retention  of  soldiers  in  their  second  enlistment.  Merging  data  from  EDMIS, 
SOC,  and  PEF  will  provide  the  data  describing  this  cohort. 

•  Cohort  3a:  Soldiers  promoted  to  E-4  in  1998.  This  cohort  consists  of  all  active  duty 
soldiers  promoted  to  E-4  in  1998.  The  primary  purpose  of  analyzing  this  cohort  is  to 
determine  whether  ACES  participation  affects  the  occurrence  and  timing  of  promotions 
to  E-5  (through  2001).  Merging  data  from  EDMIS,  AARTS,  and  PEF  will  create  the 
database  describing  this  cohort. 

•  Cohort  4a:  Soldiers  promoted  to  E-5  in  1998.  This  cohort  consists  of  all  active  duty 
soldiers  promoted  to  E-5  in  1998.  The  primary  purpose  of  analyzing  this  cohort  is  to 
determine  whether  ACES  participation  affects  the  occurrence  and  timing  of  promotions 
to  E-6  (through  2001).  One  potential  problem  with  this  analysis  is  that  ACES 


assigned  to  CONUS,  it  would  likely  over-represent  soldiers  in  a  small  number  of  military  occupations,  and  it  would 
over-represent  soldiers  assigned  to  large  installations, 

A  study  of  DoD’s  Tuition  Assistance  program  (Boesel  and  Johnson,  1988)  found  that  TA  program  participation 
rates  peaked  at  the  E-5  and  E-6  level.  At  the  time  the  data  were  collected  for  this  study,  however,  the  TA  program 
paid  for  90%  of  tuition  costs  for  soldiers  in  grades  E-5  through  E-9,  but  only  75%  of  tuition  costs  for  soldiers  below 
grade  E-5. 
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participation  is  practically  a  requirement  for  promotion  to  E-6.20  Consequently,  the 
measure  of  ACES  participation  needs  to  be  more  than  a  dichotomous  variable  indicating 
that  the  soldier  has  participated.  Recommended  measures  include  the  number  of  credit 
hours  of  college  courses  completed  and  award  of  a  two-  or  four-year  postsecondary 
degree.  Merging  data  from  EDMIS,  AARTS,  and  PEF  will  create  the  database  for  this 
cohort. 

•  Cohort  5:  Soldiers  in  the  1999  SADP.  This  cohort  consists  of  all  active  duty,  enlisted 
Army  members  who  participated  in  the  1999  SADP.  The  primary  purpose  of  analyzing 
this  cohort  is  to  determine  whether  participation  in  the  tuition  assistance  program  or  other 
rated  educational  programs  increased  the  propensity  to  reenlist  as  indicated  by  the  survey 
respondents’  intentions  to  reenlist. 

The  1999  SADP  cohort  can  be  intersected  with  three  of  the  other  four  cohorts  to  produce 
additional  cohorts  of  interest.  We  do  not  recommend  intersecting  the  SADP  with  Cohort  1, 
because  there  is  already  a  rich  set  of  attitudinal  data  collected  for  this  cohort.  Use  of  these 
intersected  cohorts  allows  the  analyst  to  consider  other  explanatory  variables  (e.g.,  satisfaction 
measures)  that  are  included  in  the  survey  but  not  in  administrative  data  sources.  Whether  these 
cohorts  provide  useful  information  will  depend  substantially  on  their  size.  The  SADP  data 
include  approximately  25,000  observations  from  Army  enlisted  personnel  and  officers;  the 
intersection  of  this  cohort  with  the  other  cohorts  will  have  substantially  fewer  observations.  The 
following  cohorts  are  obtained  by  the  intersections  of  Cohort  5  with  each  of  the  other  cohorts: 

•  Cohort  2b:  Soldiers  in  both  Cohort  2a  and  Cohort  5. 

•  Cohort  3b:  Soldiers  in  both  Cohort  3a  and  Cohort  5. 

•  Cohort  4b:  Soldiers  in  both  Cohort  4a  and  Cohort  5. 

Analysis  of  these  cohorts  would  allow  one  to  measure  the  impact  of  ACES  on  early  first- 
term  attrition  (Cohort  1),  soldier  reenlistment  (Cohorts  2a,  2b,  and  5),  promotions  to  E-5  (Cohort 
3a  and  3b)  and  to  E-6  (Cohort  4a  and  4b),  and  reclassification  (Cohort  1, 3a,  3b,  4a,  and  4b)  (see 
Table  5). 


Army  Regulation  600-8-19  (Enlisted  Promotions  and  Reductions)  states  that  10  promotion  points  are  awarded  for 
completion  of  one  of  the  following  education  improvements  while  on  active  duty:  (a)  obtains  a  high  school  diploma 
or  GED,  or  (b)  completes  any  postsecondary  course  or  test.  Furthermore,  if  recommended  for  SSG  [E-6],  the  soldier 
must  have  completed  one  of  the  actions  in  grade  SGT  [E-5], 
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Table  5 

Analysis  Cohorts,  ACES  Participation,  and  Possible  Impacts  on  Army  Readiness 

ACES  Programs  Analyzed 
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Overview  of  the  Methodological  Approach  and  Data  Issues 

The  purpose  of  this  section  is  to  provide  a  brief  overview  of  the  proposed  approach  for 
the  evaluation  and  to  introduce  several  methodological  and  data  issues  that  the  evaluation  must 
address.  This  overview  will  provide  the  reader  with  a  framework  for  better  understanding  the 
proposed  methods  and  models  described  in  more  detail  in  the  following  three  sections. 

Methodological  Approach 

Multivariate  regression  techniques  can  help  isolate  and  quantify  the  impact  of  ACES  on 
soldier  retention  and  job  performance  while  controlling  for  other  factors  that  potentially  affect 
retention  and  performance.21  The  techniques  proposed  here  are  consistent  with  those  used  in  the 
literature,  although  in  many  aspects  this  evaluation  will  be  more  rigorous  than  earlier  studies  of 
Department  of  Defense  (DoD)  voluntary  education  programs.22  Multivariate  regression  uses  data 
on  individual  soldiers  to  model  the  occurrence  of  an  event  based  on  the  soldiers’  attributes,  job 
characteristics,  and  participation  in  an  ACES  program. 

The  choice  of  regression  technique  depends  on  the  nature  of  the  dependent  variable,  or 
outcome  measure,  of  interest.  To  study  retention,  one  can  analyze  the  soldiers’  decision  whether 
to  stay  in  the  Army  (i.e.,  reenlistment  decisions)  and  the  timing  of  a  separation  (i.e.,  early  first- 
term  attrition).  Studies  in  the  military  literature  on  the  effects  of  voluntary  education  programs 
on  job  performance  use  promotions  and  timing  of  promotions  as  indicators  of  superior 
performance.  All  of  the  outcomes  of  interest  (i.e.,  reenlistment,  attrition,  reclassification,  and 
promotion)  are  dichotomous,  although  some  outcomes  incorporate  a  time  factor.  Logistic 
regression  (or  probit  analysis)  and  survival  analysis23  are  the  most  appropriate  techniques  for 
estimating  the  impact  of  ACES  on  soldier  retention  and  job  performance. 


21  An  alternative  to  the  multivariate  regression  approach  is  a  “matched-pairs”  analysis  in  which  soldiers  who 
participate  in  ACES  are  “matched”  to  soldiers  who  do  not  participate  in  ACES.  The  matching  process  uses 
observable  soldier  attributes  to  identify  a  counterpart  for  every  soldier  in  the  sample  who  participated  in  ACES. 
Magnum,  Magnum  and  Hanson  (1990)  evaluated  several  studies  on  worker  training  that  used  matched-group 
comparisons.  They  concluded  that  the  results  are  highly  sensitive  to  the  degree  to  which  accurate  matching  occurs 
(p.  80).  Both  a  regression  analysis  and  a  Matched-pairs  analysis  can  be  used  to  control  for  member  characteristics 
and  job  characteristics  that  have  been  hypothesized  to  affect  the  dependent  variable.  The  techniques  should  produce 
similar  findings. 

22  The  general  approach  for  the  retention  analysis  will  follow  the  approaches  used  by  Hogan  and  Smith  (1991)  to 
study  the  affects  of  the  Army  College  Fund  on  soldier  attrition  and  retention  and  by  Garcia  et  al.  (1998)  to  evaluate 
the  retention  effects  of  the  Navy’s  VOTED  program.  The  general  approach  for  the  promotion  analysis  will  follow 
the  approach  used  by  Garcia  et  al.  to  evaluate  the  effects  of  VOLED  on  the  performance  of  sailors  in  the  Navy. 

23  Other  names  for  survival  analysis  include  “hazard  rate  analysis,”  “event  history  analysis,”  “duration  analysis,”  and 
“transition  analysis.” 
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Logistic  regression  is  used  to  model  the  occurrence  of  a  dichotomous  outcome  (e.g., 
reenlists  versus  does  not  reenlist,  or  promoted  versus  not  promoted)  as  a  function  of  certain 
explanatory  variables.24  (An  alternative  to  logistic  regression  is  a  probit  analysis,  although  both 
modeling  techniques  should  result  in  similar  findings). 

In  survival  analysis,  the  key  variable  of  interest  is  the  length  of  time  until  a  certain  event 
occurs — e.g.,  separation  or  promotion.  In  survival  analysis,  a  “spell”  refers  to  the  length  of  time 
until  the  occurrence  of  the  event.  For  example,  to  study  early  first-term  attrition,  the  spell  would 
be  the  length  of  time  between  accession  and  separation  (for  those  who  separate  early)  or  between 
accession  and  either  contract  completion  or  the  end  of  the  data  collection  period.  To  study 
promotions  from,  say,  E4  to  E5,  the  start  date  would  be  the  date  promoted  to  E4  and  the  end  date 
would  be  the  date  promoted  to  E5  (for  those  promoted)  or  either  the  separation  date  or  the  end  of 
data  collection  period  (for  those  not  promoted).  Thus,  the  spell  is  defined  as  the  length  of  time 
between  promotion  from  E4  to  E5,  or  the  length  of  time  from  E4  to  when  the  observation  is 
censored  (i.e.,  the  soldiers’  separation  date  or  the  end  of  the  analysis  period). 

Each  spell  is  represented  by  a  number  T,  which  has  a  distribution  over  time  (t)  and  is  a 
function  of  the  hypothesized  or  observed  explanatory  variables  that  affect  T.25  One  statistical 


24 

The  logit  form  equation  to  predict  the  event  (£)  as  a  function  of  explanatory  variables  (X)  and  parameters  ((3)  is 
expressed: 

Prob(£  =  l|X)  = - —K - , 

1  +  e 

,  „  f  1  if  event  occurred,  and 

where  E  =  < 

[0  if  event  did  not  occur 

25 

‘  Suppose  that  T  has  a  continuous  probability  distribution  fit)  where  t  is  the  realization  of  T.  The  cumulative 
probability  (for  a  given  soldier)  that  the  event  occurs  is: 


F(t)  =  J  f(s)ds  =  Prob(  T  <  t) 

j=Q 


The  probability  that  a  spell  is  at  least  t  in  length  is  given  by  the  survival  function: 

SCO  =  1-F(/)  =  ProbCT  >  t). 

Given  that  the  event  has  not  occurred  prior  to  time  t,  one  is  interested  in  the  probability  that  the  event  will  occur 
during  the  next  time  period  (A).  This  can  be  expressed  mathematically  as: 

A(t,A)=Prob(f  <T<  f+A|f  <  I). 

The  likelihood  that  the  event  occurs  during  A,  as  A  becomes  infinitely  small,  is  the  instantaneous  rate  of  occurrence, 
which  is  called  the  hazard  rate.  The  hazard  rate  can  be  expressed  in  terms  of  the  probability  distribution  and  the 
survival  function: 
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issue  that  can  be  addressed  by  survival  analysis  is  data  censoring.  Some  individuals  might  leave 
the  sample  before  the  event  of  interest  occurs,  and  failure  to  address  censoring  might  lead  to 
incorrect  findings.  For  example,  in  an  analysis  of  promotions,  some  soldiers  will  separate  from 
the  military  prior  to  being  promoted.  Censoring  is  discussed  in  more  detail  below. 

Data  and  Methodological  Issues 

Several  data  and  methodological  issues  increase  the  complexity  of  the  evaluation  and  its 
ability  to  provide  precise  and  unbiased  estimates  of  the  retention  and  job  performance  benefits  of 
ACES  participation.  One  issue  is  that  soldiers  who  participate  in  ACES  (i.e.,  the  “test”  group) 
are  not  a  random  sample  drawn  from  the  population  of  soldiers.  Some  soldier  attributes  that 
influence  the  soldier’s  propensity  to  participate  in  ACES  might  also  influence  soldier  retention 
and  job  performance.  A  second  issue  is  censoring,  when  either  ACES  participation  or  the 
retention  or  promotion  outcome  of  interest  occurs  outside  the  period  over  which  soldiers  are 
observed.  A  third  issue  is  the  handling  of  missing  data.  A  fourth  issue  is  measurement  error.  A 
fifth  issue,  which  is  relevant  to  an  analysis  of  early  first-term  attrition,  is  unobserved 
heterogeneity. 

Non-random  Assignment/Self-Selection.  The  ideal  experimental  design  to  evaluate  ACES, 
as  discussed  previously,  would  be  a  controlled  experiment  where  soldiers  were  randomly 
assigned  to  a  test  group  that  was  eligible  to  participate  in  ACES  or  to  a  control  group  that  was 
not  eligible  to  participate  in  ACES.  Then,  each  group  would  be  observed  over  time  to  determine 
if  the  retention  behavior  and  job  performance  of  the  two  groups  differed  systematically.  Random 
assignment  to  a  test  and  control  group  is  not  possible  for  this  evaluation,  however,  as  is  the  case 
for  many  studies  of  human  behavior.  6 

The  methodological  problem  that  arises  because  of  non-random  participation  in  ACES  is 
that  many  soldier  attributes  (e.g.,  motivation  and  ability)  that  increase  the  propensity  to 
participate  in  ACES  likely  are  correlated  with  retention  and  job  performance.  Similarly,  some 


h(t)  = 


fit) 

Sit)’ 


which  generally  is  preferable  to  model  than  either  the  probability  distribution  or  the  survival  function.  Three 
distributions  of  T  that  are  common  in  survival  analysis  are  the  exponential,  Gompertz,  and  Weibull  distributions. 
The  models  that  use  these  distributions  are  known  as  the  exponential,  Gompertz,  and  Weibull  models,  respectively, 
and  are  members  of  the  general  class  known  as  proportional  hazards  models.  The  exponential  model  assumes  that' 
the  hazard  is  constant  over  time— i.e.,  hj(t)=Ao,  while  the  Gompertz  and  Weibull  allow  for  the  hazard  to  vary  with 
time.  When  T  follows  a  Gompertz  or  a  Weibull  distribution  the  hazard  functions  are  expressed,  respectively, 
h(r)  =  exp(p+ar)  and  h(r)  =  exp(p+a  log  t). 

26  Non-random  assignment  to  certain  military  occupations  might  also  bias  the  findings.  For  example,  if  recruits  at 
higher  risk  of  attrition  (e.g.,  non-high  school  graduates)  are  more  likely  to  be  assigned  to  an  MOS  with  lower 
training  costs,  then  the  model  might  erroneously  attribute  some  of  the  attrition  causality  to  MOS.  Likewise, 
promotion  opportunities  at  higher  ranks  will  vary  by  MOS.  One  method  to  determine  whether  non-random 
assignment  to  an  MOS  potentially  biases  the  findings  is  to  is  determine  whether  explanatory  variables  that  predict 
the  outcome  of  interest  are  also  correlated  with  the  choice  of  MOS.  A  correlation  analysis  will  be  completed  as  part 
of  the  evaluation. 
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soldiers  might  participate  in  ACES  because  they  plan  to  stay  in  the  military.  Unless  one  controls 
for  sample  selection,  the  impact  of  ACES  on  reenlistment,  attrition,  reclassification,  and 
promotion  might  be  overestimated. 

Many  of  the  characteristics  that  affect  the  decision  to  participate  in  ACES  are 
unobservable  to  the  evaluator.  That  is,  there  is  unobserved  “heterogeneity,”  or  differences, 
between  the  test  and  control  group.  Thus,  it  is  difficult  to  determine  what  differences  in 
outcomes  between  the  test  and  control  groups  should  be  attributed  to  ACES  and  what  differences 
in  outcomes  should  be  attributed  to  unobserved  heterogeneity.  As  was  stated  previously,  there 
are  four  approaches  that  can  be  used,  either  alone  or  in  combination,  to  mitigate  the  problem  of 
selectivity  bias  in  the  estimation  of  the  program  impact. 

•  The  first  approach  is  to  control  statistically  for  all  observable  soldier  characteristics  that 
might  correlate  with  either  program  participation  or  the  outcome  of  interest  by  including 
them  in  the  relevant  estimation  equations. 

•  A  second  approach  is  to  construct  an  “instrumental  variable”  for  program  participation  to 
indicate  the  treatment  effect,  rather  than  actual  participation.  To  do  this,  at  least  one 
variable  or  factor  must  be  found  that  affects  program  participation,  but  does  not  directly 
affect  outcomes.  If,  for  example,  some  assignments  (e.g.,  Korea)  make  it  more  difficult  to 
participate  in  ACES  but  do  not  affect  retention  or  promotions,  then  an  assignment 
variable  might  serve  as  an  instrumental  variable.  Intuitively,  instead  of  actual 
participation,  one  would  use  predicted  participation,  where  predicted  participation  is 
based  on  the  correlation  of  actual  participation  and  the  instrument — in  this  example,  an 
indicator  of  assignment.  Needless  to  say,  the  difficulty  with  this  method  is  discovering  a 
valid  “instrument” — the  variable  correlated  with  ACES  participation  but  uncorrelated 
with  outcomes. 

One  variation  of  the  instrumental  variable  approach  is  a  technique  known  as  the 
“Heckman  two-step  procedure”.  Using  this  technique,  the  researcher  can  calculate  a 
measure  known  as  the  “Inverse  Mills  Ratio”  that  is  used  to  control  for  unobservable 
characteristics  that  affect  both  ACES  participation  and  the  outcome  of  interest.  Garcia  et 
al  (1998)  used  the  Heckman  two-step  approach  to  control  for  self-selection  to  evaluate 
the  impact  of  the  Navy’s  Voluntary  Education  (VOLED)  Program  on  retention  and 
promotions.  Talaga  (1994)  used  this  approach  to  evaluate  the  impact  of  the  Navy’s  fully 
funded  graduate  education  program  on  Surface  Warfare  officer  promotions. 

The  first  step  in  this  technique  is  to  fit  a  probit  model  to  estimate  the  effect  of  ACES 
participation  determinants  (Z)  on  the  decision  to  participate  in  ACES.27  The  second  step 


27 


The  prohit  model  is  specified: 


Prob(  ACES  =  1 1 Z)  =  l Q#(z), 
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is  to  estimate  a  model  that  predicts  the  dependent  variable  as  a  function  of  its  explanatory 
variables.  For  example,  a  logit  model  could  be  estimated  to  predict  reenlistment  (R)  as  a 
function  of  its  determinants  ( X)  and  ACES  participation  (A)  while  controlling  for  ACES 
self-selection  via  including  the  Inverse  Mills  Ratio  (I)  as  a  control  variable  in  the 
regression,28 

Table  6  lists  explanatory  variables  contained  in  the  selected  data  sources  that  could  be 
used  in  the  first  step  of  the  Heckman  procedure  (i.e.,  the  probit  analysis)  to  predict  ACES 
participation.  Most  of  the  variables  described  in  Table  6  came  from  the  literature  review 
and  were  reported  in  Table  1.  Others  elaborate  on  these  variables.  For  example,  past 
research  shows  marital  status  to  be  a  factor  in  predicting  CE  participation.  Table  6 
includes  marital  status  in  addition  to  two  related  variables:  (a)  whether  the  soldier  has  a 
military  spouse  and  (b)  the  number  of  dependents.  Other  variables,  such  as  access  to 
ACES  programs,  were  not  addressed  in  previous  research.  These  were  selected  based  on 
a  review  of  the  available  data  sources.  For  example,  one  item  in  the  SADP  asked  whether 
the  soldier  had  another  job  during  off-duty  time.  Since  such  a  job  would  severely  restrict 
that  soldier’s  access  to  ACES  programs,  this  variable  would  likely  predict  ACES 
participation. 

•  A  third  approach  is  to  estimate  a  “fixed-effects”  model  in  which  the  soldier  serves  as  his 
or  her  own  “control.”  For  this  approach,  the  outcome  measure  (e.g.,  performance)  for 
each  soldier  is  examined  both  before  and  after  ACES  program  participation.  This 
approach  is  of  limited  use  for  this  evaluation,  though,  because  it  requires  repeated 
observations  of  the  outcome  of  interest,  and  no  standardized  measure  of  job  performance 
is  available  with  repeated  observations.  Also,  this  approach  does  not  apply  to  an  analysis 
of  retention. 


where  ACES  =  1  if  the  soldier  participated  in  ACES,  and  0  if  the  soldier  did  not  participate  in  ACES.  In  addition,  p 
is  a  vector  of  parameters  to  be  estimated.  The  Inverse  Mills  Ratio  (I)  is  calculated  from  the  normal  probability 

distribution  function,  and  the  normal  cumulative  density  function,  of  this  probit  model: 


/  - 

'  «K®,) 


28 


This  logit  model  is  expressed: 


Prob(R  =  l|X,/)  = - j - 

l  +  e  l” 

where  R  -  1  if  the  soldier  reenlisted,  and  0  if  the  soldier  did  not  reenlist. 
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Table  6 

Explanatory  Variables  to  Predict  Participation  in  ACES  Program 


Explanatory  Variables 


Academic  Achievement/  ability  at  Accession 


*  AFQT  percentile _ 

*  Have  GED _ 

*  High  school  diploma _ 

*  Some  college 


Demographic  Characteristics 


•  Age  at  accession _ 

•  Sex _ 

•  Marital  Status _ 

•  Military  spouse 


•  Number  of  dependents 


Career _ 

•  Occupation _ 

»  Time  in  service _ 

•  Current  contract  obligation  length 


•  Career  intentions  when  first  enlisted 


•  Education  aspirations  when  first  enlisted 


Satisfaction  With  Military  Lifestyle _ 

•  Measure  of  time  spent  on  assignments 
away  from  home 


•  Other  measures  of  satisfaction 


•  Intention  to  reenlist 


Enlistment  Contract _ 

*  Army  College  Fund  (ACF)  “kickers” 

•  Montgomery  G.  I.  Bill 


Access  to  ACES  Programs 


•  Assignment  outside  the  Continental 

United  States  (OCONUS) _ 

•  Installation  has  Education/Leaming 
Center 


Survey  Question 
in  the  1999  SADP 


NA 

Q105 

Q105 

Q105 


NA 


Q101 

Q54 

Q55 

Q103,  Q104 


Q59 


NA 

quo _ 

Q29,  Q30,  Q36 
Q22 


Q23 


Q14-Q17,  Q21 


Q40-Q43,  Q51 


Q32 


NA 

Q23-Q25 


Administrative  Data 
Source 


PEF  or  Project  First-Term 


PEF  or  Project  First-Term 
PEF  or  Project  First-Term 
PEF  or  Project  First-Term 


PEF  or  Project  First-Term 


PEF  or  Project  First-Term 
PEF  or  Project  First-Term 
PEF  or  Project  First-Term 
PEF  or  Project  First-Term 


PEF  or  Project  First-Term 


PEF  or  Project  First-Term 
PEF  or  Project  First-Term 
PEF  or  Project  First-Term 


NA 

NA 


Model  eligibility  criteria 
DEERS _ 


PEF  or  Project  First-Term 

PEF  or  Project  First-Term, 
combined  with  EDMIS 
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•  A  fourth  approach  is  to  examine  ACES  program  history  to  identify  “natural  experiments” 
caused  by  changes  in  program  eligibility  requirements.  Program  effects  would  be 
measured  based  not  necessarily  on  actual  participation,  but  on  the  opportunity  to 
participate.  The  measured  impact  would  be  the  impact  of  the  program  on  the  soldiers  who 
had  access  to  the  program  compared  to  soldiers  who  did  not,  after  controlling  for  other 
differences  between  the  two  groups  that  might  potentially  affect  outcomes.  Because  the 
individual’s  actual  choice  to  participate  or  not  is  not  used  as  the  treatment  indicator, 
potential  self-selection  bias  is  reduced. 

The  problem  with  self-selection  is  that,  because  it  is  an  unobservable  variable,  one  is 
never  certain  of  the  extent  to  which  the  control  variables  have  successfully  captured  the  influence 
of  the  unobservables  on  the  estimated  relationship  between  ACES  participation  and  the 
dependent  variable. 

Censored  data.  This  evaluation  will  analyze  data  on  soldier  participation  in  ACES  and 
various  soldier  outcomes  that  occurred  during  a  specified  period  of  time  (e.g.,  between  1999  and 
2001).  Because  the  observation  period  does  not  cover  the  full  military  career  of  some  soldiers, 
the  researcher  does  not  have  knowledge  of  events  that  occurred  prior  to  1999  (i.e.,  “left” 
censoring)  or  after  2001  (i.e.,  “right”  censoring).  Sample  attrition  might  cause  additional  right 
censoring.  For  example,  in  an  analysis  of  promotions,  some  soldiers  might  separate  from  the 
Army  during  the  observation  period  (e.g,,  for  medical  reasons).  Thus,  the  researcher  does  not 
observe  a  promotion  that  might  have  occurred  had  the  soldier  remained  in  the  Army.  There  are 
two  distinct  censoring  issues  relevant  to  this  evaluation — censoring  of  ACES  participation,  and 
censoring  of  the  soldier  outcomes  of  interest. 

As  discussed  previously,  electronic  data  on  ACES  participation  prior  to  1999  are  not 
available  for  some  soldiers.  That  is,  ACES  participation  data  are  left  censored  for  some  soldiers. 
Censoring  is  likely  to  be  a  greater  problem  for  soldiers  who  have  more  time  in  service  or  were 
assigned  overseas  prior  to  1999.  Time  in  service  and  overseas  assignment  both  might  be 
correlated  with  soldier  reenlistment  behavior  and  job  performance.  Consequently,  even  though 
one  can  identify  soldiers  for  whom  there  might  be  incomplete  ACES  participation  data,  simply 
omitting  these  soldiers  from  the  analysis  could  introduce  bias  in  the  findings. 

Censoring  means  that  ACES  participation  data  will  be  measured  with  some  degree  of 
inaccuracy  depending  on  how  participation  is  measured.  Measures  of  recent  participation  will 
not  be  censored,  while  measures  of  “ever”  participated  will  be  censored  for  some  soldiers.  The 
likely  result  is  that  the  estimated  relationship  between  the  censored  variable  “ever  participated” 
in  ACES  and  the  outcome  of  interest  will  be  biased  towards  zero  (that  is,  no  effect).  The  cohorts 
discussed  previously  were  chosen  to  minimize  the  problem  with  censoring.  For  Cohort  1, 
accessions  in  FY  1999,  there  should  be  very  little  censoring.29  For  cohorts  2  through  4,  however, 
only  data  on  ACES  participation  during  the  most  current  three  years  (i.e.,  1999  through  2001) 
will  be  available  producing  left  censoring  for  these  cohorts. 


EDMIS  participation  data  will  be  missing  before  January  1999  for  a  small  number  of  sites  that  would  represent 
very  unlikely  assignments  for  soldiers  in  their  first  three  months  of  service.  Furthermore,  these  soldiers  are  in 
training,  and  not  likely  to  participate  in  ACES  programs. 
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Censoring  is  also  a  problem  with  some  of  the  soldier  outcomes  of  interest — especially 
when  the  outcome  is  related  to  the  timing  of  an  event  (e.g.,  timing  of  promotions  or  timing  of 
separations).  For  example,  as  noted  above,  in  a  study  of  whether  ACES  participation  affects  the 
timing  of  promotions,  some  soldiers  will  separate  from  the  Army  and  thus  not  realize  a 
promotion  that  would  have  occurred  had  the  soldier  remained.  Survival  analysis  allows  one  to 
control  for  censoring  of  the  dependent  variable. 

Missing  data.  Data  on  the  analysis  cohort  might  be  missing  for  several  reasons.  First,  a 
particular  measure  of  a  soldier  characteristic  might  not  be  collected  or  might  be  unobservable  to 
the  researcher.  Second,  poor  record  collection  or  management  could  result  in  missing  values  for 
selected  variables.  Third,  data  in  administrative  files  might  be  purged  or  overwritten  because  of 
storage  limitations. 

The  first  reason  can  complicate  the  analysis  in  two  ways.  As  discussed  previously, 
unobserved  characteristics  that  are  correlated  with  both  the  outcomes  of  interest  and  ACES 
participation  (or  other  explanatory  variables  in  the  model)  could  bias  the  findings  (i.e.,  omitted 
variable  bias).  In  an  analysis  of  attrition,  if  the  missing  characteristic  is  correlated  with  timing  of 
attrition  but  not  correlated  with  the  explanatory  variables  in  the  model  then  the  baseline  hazard 
rates  might  be  biased  (i.e.,  unobserved  heterogeneity  bias).  Unobserved  heterogeneity  is 
discussed  later. 

Poor  data  collection  or  management  could  result  in  incomplete  data  on  soldiers.  Although 
EDMIS  has  helped  to  standardize  the  ACES  data  collection  process,  there  still  might  be  variation 
across  installations  in  the  completeness  and  accuracy  of  soldier  records  on  participation  in 
various  ACES  programs.30  The  information  we  have  obtained  from  PERSCOM  indicates  that 
there  is  variation  across  installations  in  the  accuracy  of  recording  some  EDMIS  variables.  The 
extent  of  the  problem  will  not  become  apparent  until  the  data  collection  process. 

The  extent  of  the  missing  data  problems  will  not  be  apparent  until  the  data  have  been 
collected  and  preliminary  analyses  performed.  If  there  does  appear  to  be  a  problem  with  missing 
data,  then  appropriate  statistical  techniques  to  minimize  the  problem  can  be  identified  and 
applied. 

Measurement  error.  Measurement  error  occurs  when  precise  measures  of  a  particular 
variable  of  interest  are  not  available.  The  biases  introduced  by  measurement  error  can  be  severe. 
For  this  analysis,  measurement  error  is  a  potential  problem  for  two  key  variables — ACES 
program  participation  and  soldier  job  performance. 

•  Alternative  measures  of  ACES  participation  differ  in  their  ability  to  capture  the  degree  to 
which  soldiers  use  ACES  services.  For  example,  participation  in  the  tuition  assistance 
program  could  be  measured  using  (a)  a  dichotomous  measure  of  ever  having  participated 
in  the  program,  (b)  number  of  credit  hours  taken,  (c)  number  of  courses  taken,  and  (d)  type 


30  A  previous  study  of  ACES  (Brink  et  al.,  1981)  found  missing  ACES  participation  data  to  be  a  common 
phenomenon.  This  study,  though,  predates  EDMIS  and  collected  information  on  ACES  participation  by  extracting 
the  data  from  soldier  personnel  files. 
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of  courses  taken.  Furthermore,  some  courses  might  be  directly  relevant  to  the  soldiers’ 
current  job,  while  other  courses  might  have  little  relevance  to  the  soldiers’  current  job. 

•  Little  information  on  job  performance  is  systematically  collected  in  the  Army  and  stored  in 
soldiers’  personnel  records.  Measures  that  are  collected  typically  are  not  comprehensive, 
but  deal  with  specific  requirements  (such  as  marksmanship  and  level  of  physical  fitness)  or 
pertain  to  specific  military  occupational  specialties  (such  as  scores  on  proficiency  tests). 

For  this  reason,  researchers  often  use  promotions  (and  demotions)  as  an  indicator  of 
superior  (inferior)  performance.  Both  the  occurrence  of  promotions  and  how  quickly  they 
occur  are  indicators  of  performance.  Promotions,  though,  are  imprecise  measures  of  job 
performance,  and  factors  other  than  performance  that  affect  promotions  (e.g.,  time 
requirements)  decrease  the  precision  of  promotions  and  timing  of  promotions  as  indicators 
of  job  performance. 

Unobserved  heterogeneity  versus  state  dependence.  Unobserved  heterogeneity  is  the 
name  given  in  the  economics  literature  to  the  unobserved,  individual-specific  factors  affecting 
behavior.  The  issue  of  self-selection  in  ACES  participation  discussed  above  is  an  unobserved 
heterogeneity  issue.  A  different,  but  related,  issue  that  is  especially  relevant  to  the  study  of 
attrition  is  the  effect  of  unobserved  heterogeneity  on  when  members  separate  from  the  Army.  If 
unobserved  differences  across  members  affect  whether  they  separate  prior  to  contract  completion 
and  the  timing  of  separations,  it  might  be  difficult  to  discern  whether  attrition  rates  change  over 
time  due  to  tenure,  or  whether  members  with  higher  probabilities  of  attrition  are  systematically 
selected  out  of  the  Army.31 

Consider  the  following  example.  Soldiers  with  less  taste  for  military  life  are  more  likely 
to  separate  by  choice  early  during  their  enlistment  so  that  over  time  the  remaining  sample  is 
increasingly  made  of  up  of  soldiers  with  a  higher  taste  for  military  life.  Thus,  initially  the 
estimated  hazard  rate  reflects  the  average  of  the  hazard  rates  for  all  soldiers,  but  over  time  the 
hazard  rate  becomes  more  like  that  of  the  soldiers  with  higher  taste  for  military  life.32 

Likewise,  the  hazard  rate  is  a  function  of  the  state  of  events  that  take  place  during  the 
enlistment  period.  The  state  of  events  differs  during  basic  training,  advanced  individual  trainings 
and  the  post-training  environment.  Military  life  is,  arguably,  the  most  restrictive  during  basic 
training,  followed  by  advanced  training,  followed  by  the  post-training  environment. 
Consequently,  soldiers  are  more  likely  to  separate  by  choice  during  basic  training,  followed  by 
advanced  training,  followed  by  the  post-training  environment.  Thus,  it  might  be  difficult  to 
determine  if  declining  attrition  hazard  rates  in  the  sample  are  due  to  unobserved  heterogeneity,  or 
due  to  the  nature  of  the  enlistment  lifecycle  itself  and,  in  particular,  the  increasing  attractiveness 


31  Another  variant  of  unobserved  heterogeneity  and  state  dependence  is  the  ability  to  distinguish  between  cohort  and 
time  effects.  The  only  source  of  variation  in  some  explanatory  variables  (e.g.,  economic  conditions)  is  over  time. 
Trends  in  economic  conditions  might  correspond  with  the  natural  decline  in  the  attrition  rate  over  the  course  of  the 
first-term  enlistment,  thus  masking  the  true  relationship  between  attrition  and  its  explanatory  variables. 

32  The  hazard  rate  is  the  likelihood  of  separation  between  time  period  t  and  t+1,  given  that  separation  has  not 
occurred  prior  to  t  (see  footnote  #25). 
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of  Army  life  as  soldiers  move  through  and  complete  the  training  process.  This  is  a  concern 
because  soldiers  are  more  likely  to  participate  in  various  ACES  programs  at  different  stages  of 
their  career.  During  the  first  part  of  the  enlistment  when  attrition  is  relatively  high,  soldiers 
typically  do  not  participate  in  ACES.  During  the  later  part  of  the  enlistment  when  attrition  is 
lower,  soldiers  will  have  had  greater  opportunity  to  participate  in  ACES.  Consequently,  without 
controlling  for  unobserved  heterogeneity  one  might  overestimate  any  effect  that  ACES  has  on 
reducing  attrition. 


The  Retention  Analysis 

Why  do  soldiers  stay  in  the  Army  or  leave?  Knowing  what  factors  increase  reenlistment 
rates  and  decrease  attrition  is  of  vital  importance  to  the  Army  because  of  the  high  cost  of 
turnover  in  terms  of  workforce  planning  and  the  high  costs  to  recruit  and  train  replacements  for 
soldiers  who  leave.  Numerous  studies  in  the  military  and  civilian  literature  have  analyzed  the 
issue  of  turnover  to  identify  those  factors  that  an  organization  can  change  to  improve  retention. 
Griffeth  and  Horn  (1995),  in  their  review  of  the  civilian  literature  on  employee  turnover,  wrote 
that: 


Prescriptions  for  reducing  employee  turnover  abound... However  popular,  such  advice 
often  rests  on  dubious  or  nonexistent  empirical  underpinnings.  All  too  often,  practical 
remedies  are  derived  from  case  studies  or  anecdotal  evidence.  Rigorous  research  on 
practical  interventions — especially  those  using  quasi- experimental  or  experimental 
designs — is  remarkably  scarce  (p.  193). 

The  purpose  of  this  evaluation  plan  is  to  help  ensure  a  rigorous  evaluation.  The  proposed 
methods  and  model  specification  build  on  the  most  rigorous  studies  in  the  civilian  and  military 
literature.  A  conceptual  model  is  first  described  that  summarizes  the  hypothesized  effect  that 
ACES  program  participation  and  other  factors  have  on  soldier  retention.  Separate  conceptual 
models  are  presented  to  describe  the  reenlistment  decision  and  attrition.33  Testable  hypotheses 
are  derived  from  the  conceptual  models  that  help  to  answer  the  questions  posed  in  the 
Introduction  to  this  plan.  Finally,  an  appropriate  functional  form,  statistical  techniques,  and  data 
are  identified  to  perform  the  analyses  of  reenlistments  and  attrition. 

In  addition  to  estimating  multivariate  regression  models  to  control  for  selection  bias,  the 
evaluation  should  provide  tabulations  depicting  the  number  of  soldiers  who  participate  in  various 
ACES  programs  and  compare  retention  rates  for  soldiers  who  do  and  do  not  participate  in  the 
various  programs.  Also,  the  analysis  should  provide  separate  analyses  for  subgroups  of 
soldiers — e.g.,  by  rank,  sex,  race/ethnicity,  occupation,  and,  where  possible.  Army  career 
intentions. 


3j  Some  studies  in  the  military  literature  on  retention  model  reenlistments  and  attrition  together.  The  impact  of 
ACES  on  retention,  however,  will  likely  differ  in  its  impact  on  reenlistments  and  its  impact  on  attrition. 
Consequently,  conducting  separate  analyses  of  reenlistments  and  attrition  is  more  appropriate  than  a  combined 
analysis. 
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Reenlistment 


The  conceptual  model.  The  military  and  civilian  literature  on  employee  retention 
identifies  many  factors  that  have  been  shown,  or  are  hypothesized,  to  influence  turnover 
behavior.  To  isolate  the  effect  of  ACES  program  participation  on  reenlistments,  it  is  important  to 
identify  and  control  for  factors  that  potentially  affect  soldiers’  reenlistment  behavior. 

One  category  of  factors  is  compensation  and  benefits.  If  soldiers  perceive  that  their 
expected  earnings  (e.g.,  basic  pay,  special  pays  and  allowances,  and  retirement  pay)  and  benefits 
(e.g.,  healthcare  coverage)  from  remaining  in  the  military  are  lower  than  can  be  received  in  the 
civilian  sector,  then  soldiers  are  more  likely  to  leave  the  Services.34  A  second  category  of  factors 
is  quality-of-life  issues.  This  includes  factors  such  as  the  hardship  associated  with  frequent  or 
lengthy  deployments,  the  characteristics  of  one’s  job,  and  satisfaction  with  the  Army  lifestyle.  In 
general,  the  more  that  the  quality-of-life  factor  adds  to  the  soldiers’  level  of  satisfaction  (or 
“utility”)  the  greater  the  retention  effect  of  that  factor.  A  third  category  is  soldier  demographic 
characteristics.  Often,  these  factors  are  included  in  empirical  analyses  without  thought  for  why 
these  factors  are  expected  to  affect  the  outcome  of  interest.  Factors  such  as  race,  sex,  and  level  of 
education  might  be  indicators  of  potential  job  opportunities  and  earnings  in  the  civilian  sector. 
Factors  such  as  marital  status  and  number  of  dependents  might  reflect  the  costs  of  military 
hardship  on  the  family  (and  thus  interact  with  quality-of-military-life-issues),  or  these  factors 
might  be  proxies  for  the  services  the  military  provides  to  families. 

Generally,  no  single  factor  is  decisive  in  causing  employees  to  separate  or  reenlist,  and 
isolating  the  effect  of  ACES  on  the  reenlistment  decision  requires  the  use  of  fairly  complex 
analytical  techniques.  Consider  the  following  simplified  model  of  how  ACES  program 
participation  might  affect  a  soldier’s  decision  to  reenlist.  Let  Yit  represent  the  reenlistment 
outcome  of  individual  i  at  time  period  t.  Furthermore,  let  A,-  be  an  indicator  of  the  “treatment” — 
i.e.,  that  the  individual  soldier  has  participated  in  a  particular  ACES  program.  Then,  the  equation 
Yit=f(Xit,  Ai )  describes  the  effect  that  ACES  participation  has  on  the  soldiers’  outcome,  where  X  is 
a  vector  of  other  variables  potentially  affecting  the  outcome,  and /(...)  is  the  functional  form  of 
the  estimation  equation. 


34  Studies  of  military  retention  often  estimate  an  Annualized  Cost  of  Leaving  (ACOL)  Model  in  which  the  ACOL 
value  is  equal  to  the  annualized  difference  between  military  and  civilian  pay  over  an  optimal  time  horizon  or  period 
of  stay  in  the  military.  To  calculate  the  ACOL  variable,  assume  that  an  individual  can  stay  in  the  military  for  a 
maximum  of  “n”  more  years,  and  will  stay  in  the  labor  force  “T”  more  years,  regardless  of  when  he  or  she  leaves  the 
military.  The  ACOL  is  calculated  using  the  following  equation: 

ACOLn  =  £d‘(Mt-Wt0)+  f1d‘(Wtl:-Wl0). 

k=l  k=n+\ 

where  Mk  is  expected  military  pay  in  year  k(k=  1, 2 Wk0  is  future  potential  civilian  earnings  from  leaving 
immediately  (k  =  1,2,.. .,7);  Wkn  is  future  potential  civilian  earnings  from  staying  n  more  years,  where  civilian  wages 

are  conditional  on  n  years  of  military  experience  (k  =  n+ 1,  n+2,...,T);  dk  equals  (j ;  and  r  is  the  personal  discount 
rate. 
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Retention  is  often  modeled  using  a  random  utility  model  that  highlights  the  roles  of 
expected  utility  from  remaining  (UR)  in  the  military  versus  the  expected  utility  from  leaving 
(ir).35  One  does  not  directly  observe  the  expected  utility  of  remaining  in  the  Service.  One  does, 
however,  observe  when  the  expected  utility  of  remaining  is  greater  than  that  of  leaving.  Such  a 
positive  net  utility  for  remaining  would  be  indicated  by  the  soldier’s  choice  to  remain.36 

Hypotheses  derived  from  the  conceptual  model.  The  conceptual  model  can  be  used  to 
derive  testable  hypotheses  describing  the  effects  of  ACES  on  reenlistment  decisions.  Although 
these  expected  effects  are  derived  rationally  from  the  model,  they  are  often  consistent  with  a 
large  body  of  empirical  research.  As  Table  7  illustrates,  theory  can  provide  some  guidance  on  the 
likely  direction  of  the  relationship  between  the  decision  to  reenlist  and  its  determinants. 

However,  the  magnitude  of  the  relationships  must  be  determined  empirically.  To  the  extent  that 
ACES  services  improve  the  soldiers’  quality  of  life  and  increase  promotion  potential, 
participation  in  ACES  programs  would  be  predicted  to  improve  retention.  However,  some  ACES 
programs  might  have  contradicting  effects  on  retention.  For  example,  taking  college  classes 
funded  by  TA  can  increase  the  likelihood  of  promotion  within  the  Army,  while  it  also  increases 
the  soldier’s  earnings  potential  outside  the  military.  Thus,  this  program  could  have  mixed  effects 
on  reenlistment  rates.  Programs  that  are  operational  in  nature,  such  as  MOS  improvement 
courses,  primarily  affect  the  utility  of  staying  in  the  Army. 

The  effect  of  ACES  participation  on  retention  will  likely  differ  by  ACES  program.  ACES 
programs  that  provide  military-specific  knowledge  and  skills  (e.g.,  MOS  Improvement  Courses) 


35 


Mathematically,  the  random  utility  model  is  specified: 


The  net  utility  of  remaining  at  time  t  is  given  by: 


(t/*-C/s),=/?3T,+M  +  ;f  +  ^, 

where  X  is  the  matrix  of  explanatory  variables;  P  is  a  vector  of  parameters  that  describe  the  relationship  between  the 
explanatory  variables  and  the  net  utility  of  remaining;  X  is  a  parameter  that  describes  the  relationship  between  ACES 
program  participation  and  the  net  utility  of  remaining;  I  and  y  are,  respectively,  the  Inverse  Mills  Ratio  to  control  for 
self-selection  into  ACES  and  the  parameter  for  this  variable;  and  s  is  a  random  error  term. 

36  Thus, 


P,  =  Prob(Remaining, ) 

=  Prob(£  ( fXt  +  AA + yf  +  st )  /(I  +  >  0 1  Xt ) 

=  Prob(5  +  fi'X+XA  +  /I+e>  0|Xt). 

That  is,  the  probability  of  remaining  at  time  t  is  a  function  of  the  net  utility  of  remaining  at  time  t ,  which  is  a 
function  of  the  net  utility  of  remaining  during  each  period  after  t  (i.e,,  at  time  f+1,  t+ 2,  ..,.7)  discounted  to  the 
present  time.  This  model  can  be  estimated  using  either  logistic  regression  or  a  probit  analysis. 
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might  have  a  substantially  different  effect  on  retention  than  programs  that  provide  knowledge 
and  skills  that  are  transferable  to  the  private  sector  (e.g,,  Tuition  Assistance). 

The  following  are  examples  of  testable  hypotheses  that  can  be  derived  from  the 
conceptual  model.  These  examples  apply  to  the  tuition  assistance  program,  but  similar 
hypotheses  can  be  developed  for  each  ACES  program 

•  Participation  in  the  tuition  assistance  program  increases  the  propensity  of  soldiers  to 
reenlist.  This  hypothesis  can  be  tested  using  an  indicator  of  TA  participation. 


Table  7 

Likely  Effects  of  Education  Program  Participation  on  Reenlistment 


Consequence  of  ACES  Participation 

Expected  Effect  on 
Reenlistment  Based  on 
Theory 

The  Army  Tuition  Assistance  Program 

_  (?) 

*  Increase  earnings  potential  in  the  private  sector 

•  Increase  promotion  potential 

+ 

»  Benefit  might  raise  soldier's  commitment  to  the  Army 

+ 

The  SOCAD  Program 

_  (?)  | 

•  Increase  earnings  potential  in  the  private  sector 

*  Increase  promotion  potential 

+ 

The  FAST  Program 

(+> 

*  Make  soldier  eligible  for  reenlistment 

+ 

•  Increase  promotion  potential 

+ 

MOS  Improvement  Courses 

(+) 

*  Improve  work  satisfaction 

+ 

•  Increase  promotion  potential 

+ 

NCO  Leader  Development  Courses 

(+> 

•  Improve  work  satisfaction 

+ 

•  Increase  promotion  potential 

+ 

•  The  degree  of  participation  in  the  tuition  assistance  program  affects  the  propensity  of 
soldiers  to  reenlist.  This  hypothesis  can  be  tested  using  measures  of  TA  participation 
such  as  total  tuition  assistance  received  by  soldiers  and  total  credit  hours  earned. 

•  Tuition  assistance  has  a  greater  effect  on  retention  than  other  ACES  programs. 
Determining  whether  the  total  effect  of  TA  participation  on  the  propensity  to  reenlist  is 
statistically  greater  than  the  effect  of  other  ACES  programs  can  test  this  hypothesis. 

Model  specification  and  estimation.  Conventional  probability  modeling  techniques,  such 
as  estimating  a  logit  model,  can  be  used  to  estimate  the  conceptual  model  described  above. 
Separate  models  should  be  estimated  for  cohorts  2a,  2b,  and  5.  In  addition,  separate  models 
should  be  estimated  to  analyze  the  first  year,  second  year,  and  other  year  reenlistment  decisions. 

The  dependent  variable  for  this  analysis  is  whether  the  soldier  reenlists  (for  cohorts  2a, 
and  2b),  or  whether  the  soldier  intends  to  reenlist  (cohort  5).  The  dependent  variable  based  on 
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administrative  data  is  dichotomous  and  takes  on  the  value  of  1  if  the  soldier  reenlists,  and  0  if  the 
soldier  separates.37  The  1999  SADP  asks  soldiers  their  intention  to  reenlist,  and  soldiers  are 
asked  to  respond  on  a  five-point  scale:  very  likely,  likely,  neither  likely  nor  unlikely,  unlikely,  or 
very  unlikely.  The  survey  responses  to  this  questions  could  be  coded  as  a  dichotomous  variable 
(e.g.,  l=very  likely  or  likely  to  reenlist,  and  0=all  other  responses)  and  estimated  using  logistic 
regression.  Alternatively,  an  ordered  probit  model  could  be  estimated  that  predicts  soldiers’ 
responses  to  the  five-point  scale. 

Appropriate  explanatory  variables  to  include  in  the  regression  analysis  are  listed  in  Table 
8.  These  variables  come  primarily  from  the  earlier  research  studies  that  were  described  in  our 
literature  review  (see  Table  3).  The  measures  of  ACES  program  participation  can  be  modeled  as 
indicator  or  “dummy”  variables  that  take  on  the  value  of  1  if  the  soldier  participated  in  the 
program,  and  0  otherwise.  Alternatively,  the  level  of  participation  can  be  modeled  for  some 
programs  (e.g.,  number  of  credits  taken  through  the  tuition  assistance  program).  Not  all  the 
ACES  participation  variables  will  apply  to  each  cohort  analysis.  For  example,  participation  in 
NCO  leadership  courses  might  not  be  relevant  for  studying  the  first-term  reenlistment  decision. 

After  estimating  the  logistic  regression  model(s),  several  steps  should  be  taken  to  validate 
the  findings.  The  first  is  to  have  area  experts  review  the  findings  to  judge  whether  they  are 
plausible.  The  second  and  related  step  is  to  compare  the  findings  to  estimates  from  related 
studies.  For  example,  the  estimated  effect  of  ACES  participation  on  soldier  retention  should  be 
compared  to  the  estimated  effects  of  other  factors  (e.g.,  retention  bonuses)  on  retention  to 
determine  if  the  estimates  appear  plausible.  Finally,  by  estimating  separate  models  with  the  1999 
SADP  data  and  the  administrative  data,  one  can  compare  the  two  models  to  determine  if  the 
findings  are  consistent. 

Attrition 

The  conceptual  model.  Most  attrition  occurs  in  the  early  stages  of  the  first  enlistment 
before  the  soldier  has  the  opportunity  to  participate  in  most  ACES  programs.  Consequently,  the 
analysis  of  attrition  focuses  on  the  FAST  program  and  MOS  Improvement  Courses,  which  are 
used  by  soldiers  earlier  in  their  military  career.  Both  of  these  educational  opportunities  can 
improve  soldier  skills  that  are  required  for  effective  performance  of  their  job.  This  improvement 
might  increase  the  soldier  job  satisfaction,  and  consequently  reduce  the  propensity  of  a  soldier  to 
separate  prior  to  contract  completion. 


37  An  alternative  specification  is  to  distinguish  between  “good”  and  “bad”  turnover.  Under  this  specification,  the 
variable  would  take  on  the  value  of  1  if  a  soldier  reenlists  or  if  the  soldier  leaves  because  he  or  she  is  not  eligible  for 
reenlistment,  and  0  if  a  soldier  who  is  eligible  for  reenlistment  leaves.  Alternatively,  the  analysis  could  be  restricted 
to  those  who  are  eligible  for  reenlistment. 
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Table  8 

Potential  Explanatory  Variables  for  the  Reenlistment  Analysis 


Explanatory  Variables 

Administrative  Data  Source 

Academic  Achievement/Ability 

•  AFQT  percentile 

PEF 

•  Have  GED 

PEF 

•  High  school  diploma 

PEF 

•  Some  college 

PEF 

•  Changes  in  educational  level  since  accession 

PEF 

Demographic  Characteristics 

•  Age  at  accession 

PEF 

•  Sex 

PEF 

•  Marital  status  at  accession 

PEF 

•  Changes  in  marital  status  since  accession 

PEF  A 

•  Race/ethnicity 

PEF 

Career 

•  Occupation  at  accession 

PEF 

•  Change  in  primary  MOS  since  accession 

PEF 

Bonuses/Pay 

•  Military  pay  and  benefits  compared  to  expected 
civilian  pay  and  benefits 

ACOL  Calculation 

Enlistment  Contract 

*  ACF  “kickers” 

Probability  estimate  based  on  accession 
year,  MOS,  and  AFQT  score 

•  Montgomery  G.  I.  Bill  eligibility 

Probability  estimate  based  on  AFQT  score 

•  Length  of  current  enlistment  obligation 

PEF 

Economics  Factors 

•  Unemployment  rate  at  ETS  date  in  state  where 
soldier  is  legal  resident 

PEF  &  Department  of  Labor 

Training  Phases  Completed 

•  Basic  training 

PEF  1 

•  Advanced  Individual  Training 

PEF 

Education  Program  Participation  While  On  Active 
Duty 

•  Tuition  Assistance 

EDMIS 

•  SOCAD 

SOCAD  Agreement  Databases 

•  FAST 

EDMIS 

•  MOS  Improvement  Courses 

EDMIS 

•  NCO  Leader  Development  Courses 

EDMIS 
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One  can  model  voluntary  attrition  using  a  random  utility  model  similar  to  that  formulated 
by  Hogan  (1979)  and  Hogan,  Smith  and  Sylwester  (1991).38  This  model  highlights  the  roles  of 
expected  utility  from  remaining  in  the  military  versus  leaving,  the  enlistment  contract  length,  and 
the  “cost”  of  breaking  the  enlistment  contract.  If  the  net  benefits  of  remaining  in  the  military  at 
time  t  exceed  the  costs  of  remaining  in  the  military — including  the  opportunity  cost  of  working 
in  the  civilian  sector — then  the  member  remains  in  the  military.39 

Neither  the  net  utility  of  remaining  in  the  service  nor  the  cost  to  the  soldier  of  separating 
are  directly  observed.  One  does  observe,  however,  by  soldiers’  choice  to  remain  or  leave, 
whether  the  cost  of  remaining  outweighs  the  cost  of  leaving.40  Policies  that  raise  the  cost  to  the 
soldier  of  separating,  (i.e.,  that  increase  Ct)  will  reduce  attrition. 


38 

Hogan,  Smith  and  Sylwester  (1991)  model  the  effect  of  the  Army  College  Fund  on  attrition  and  reenlistment. 
They  find  that  supplemental  education  benefits  have  only  a  small  effect  on  contract  completion. 

39  Consider  the  random  utility  model  where  Uf  denotes  a  soldier’s  dollar  value  of  utility  of  remaining  in  the 

military  at  time  t,  and  (7 /'  denotes  the  dollar-equivalent  utility  in  period  t  from  leaving.  Thus,  the  net  utility  of 
remaining  at  time  t  is  given  by: 

(UR-UL)t=/3’Xt+M+7I+sn 

where  X  is  a  matrix  of  explanatory  variables;  A  is  a  measure  of  ACES  participation;  I  is  the  Inverse  Mills  Ratio;  3, 
K,  and  y  are  parameters  that  describe  the  relationship  between  the  explanatory  variables  and  the  net  utility  of 
remaining;  and  s  is  a  random  error  term. 

Let  C£  be  the  cost  at  time  t  of  breaking  the  enlistment  contract.  The  cost  of  breaking  the  enlistment  contract  could 
include  the  negative  utility  associated  with  a  bad  discharge,  or  the  difficulties  of  securing  an  early  separation.  Then, 
at  time  tf,  an  individual  will  break  the  enlistment  contract  if: 

+  +£,)/(!+  jy-'  < c, , 

t=f 

where  T  is  the  expiration  of  the  contract. 

Note  the  implications  of  the  model.  The  soldier  has  an  incentive  to  break  the  contract  if  the  utility  of  remaining  is 
lower  then  the  utility  of  separating.  To  the  extent  that  the  ACES  program  can  improve  the  utility  of  staying,  it  can 
reduce  attrition.  Also,  the  longer  the  soldier  must  endure  a  negative  (TX — i.e.,  the  greater  the  remaining  term  of 
service — the  more  likely  that  soldier  is  to  separate. 

40  Thus, 

Pt  =  Prob(Separatingf) 

=  Prob(£  (fi’X,  +  AA  +  yl  +  st )  /(l  +  j)!"'  <  C  |  X, ) 

t=tf 

T 

=  Prob( £ {fi’X,  +A  +  yI +  £,)!{  1  +  j)M'  - C  < 0 1  Xt ) 

t=t’ 

=  Prob  (a  +  ft'X  +A  +  ]d  +  £r-C<0|Xr). 
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Hypotheses  derived  from  the  conceptual  model.  Testable  hypotheses,  derived  from  the 
causal  model  described  above,  provide  insight  as  to  the  likely  effects  of  ACES  participation  on 
attrition.  Factors  that  increase  the  utility  of  remaining  in  the  military  might  decrease  the 
likelihood  of  attrition.  Likewise,  factors  that  increase  the  utility  of  “civilian”  life  might  increase 
the  likelihood  of  attrition. 

This  study  would  have  the  ability  to  empirically  test  several  hypotheses  regarding  the 
relationship  between  ACES  participation  and  attrition.  The  following  are  examples  of  hypotheses 
that  can  be  tested. 

•  Participation  in  the  FAST  program  reduces  attrition  for  eligible  soldiers 

•  Participation  in  MOS  Improvement  Courses  reduces  attrition. 

•  The  impact  of  ACES  on  attrition  differs  by  soldier  characteristics  (e.g.,  Army  career 

intentions,  or  whether  possessed  a  high  school  degree  at  the  time  of  accession). 

Additional,  compounding  factors  that  affect  the  decision  to  separate  might  make  it 
difficult  to  isolate  the  effect  that  ACES  has  on  attrition.  Klein,  Dawson  and  Martin  (1991)  found 
that  most  recruits  who  leave  prior  to  completing  the  first  35  months  of  their  enlistment  do  so  for 
a  combination  of  two  or  more  reasons.  The  most  common  reasons  for  early  separation  are  work 
or  duty  problems,  training  problems,  minor  offenses,  and  mental  and  health  problems.  Often 
these  problems  are  interrelated  and  confounded  by  problems  with  drug  and  alcohol  abuse  or  a 
negative  attitude.  While  it  seems  reasonable  to  hypothesize  that  work  and  training  problems  can 
be  ameliorated  by  participation  in  ACES  programs,  such  as  FAST  and  MOS  Improvement 
Courses,  other  reasons  for  attrition  will  be  relatively  unaffected  by  ACES  participation. 

Model  specification  and  estimation.  The  conceptual  model  can  be  estimated  using 
conventional  probability  modeling  techniques  such  as  logistic  regression  (to  model  whether  a 
soldier  separates  prior  to  contract  completion)  and  survival  analysis  (to  model  both  the 
occurrence  and  timing  of  events).  Possible  explanatory  variables  to  include  in  the  attrition 
analysis  are  listed  in  Table  9.  Our  review  of  the  research  literature  addressing  attrition,  presented 
in  the  previous  section,  lists  some  of  the  studies  in  which  the  effects  of  these  variables  on 
attrition  were  established  (see  p.  22).  These  variables  are  also  similar  to  (or  variations  of) 
explanatory  variables  used  in  other  attrition  studies  (e.g.,  Hogan,  Smith  and  Sylwester,  1991; 
Klein  and  Martin,  1991)  or  in  the  empirical  literature  on  retention  (see  Table  3). 

Some  of  these  explanatory  variables  could  be  modeled  as  interaction  terms.  For  example, 
Klein  and  Martin  (1991)  modeled  the  interaction  of  race/ethnicity  with  AFQT  score,  and  with 
age  in  their  study  of  first-term  attrition. 
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Table  9 

Potential  Explanatory  Variables  for  the  Attrition  Analysis 


Explanatory  Variables 

Administrative  Data  Source 

Academic  Achievement/Ability 

•  AFQT  percentile 

Project  First-Term  Database 

•  Have  GED 

Project  First-Term  Database 

•  High  school  diploma 

Project  First-Term  Database 

•  Some  college 

Project  First-Term  Database 

•  Changes  in  education  level 

Project  First-Term  Database 

Demographic  Characteristics 

•  Age  at  accession 

Project  First-Term  Database 

•  Sex 

Project  First-Term  Database 

•  Marital  status  at  accession 

Project  First-Term  Database 

•  Changes  in  marital  status  since  accession 

Project  First-Term  Database 

•  Race/ethnicity 

Project  First-Term  Database 

Career 

•  Occupation  at  accession 

Project  First-Term  Database 

•  Change  in  primary  MOS  since  accession 

Project  First-Term  Database 

Bonuses/Pay 

•  Military  pay  and  benefits  compared  to  expected 
civilian  pay  and  benefits 

ACOL  Calculation 

Enlistment  Contract 

•  ACF  “kickers” 

Probability  estimate  based  on  accession 
year,  MOS,  and  AFQT  score 

•  Montgomery  G.  I,  Bill  eligibility 

Probability  estimate  based  on  AFQT  score 

•  Length  of  current  enlistment  obligation 

Project  First-Term  Database 

Economics  Factors 

•  Unemployment  rate  at  accession  date  in  state 
where  soldier  is  legal  resident 

Project  First-Term  Database  & 

Department  of  Labor 

Training  Phases  Completed 

•  BASIC  training 

Project  First-Term  Database 

•  Advanced  Individual  Training 

Project  First-Term  Database 

Education  Program  Participation  While  On  Active 
Duty 

•  The  FAST  Program 

EDMIS 

•  MOS  Improvement  Courses 

EDMIS 
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The  Performance  Analysis 


Numerous  studies  in  the  civilian  literature  have  shown  that  additional  academic  training 
improves  job  performance.  Two  challenging  issues  for  this  evaluation  are  (a)  how  to  measure  job 
performance,  and  (b)  how  to  isolate  the  impact  of  ACES  program  participation  on  job 
performance.  Because  general  job  performance  measures  are  not  collected  systematically  by  the 
Army  to  be  included  in  administrative  records,  the  evaluation  study  must  use  surrogate  measures, 
such  as  MOS  reclassification  and  the  occurrence  and  timing  of  promotions.45 

As  discussed  previously,  promotions  are  an  imprecise  measure  of  job  performance 
because  additional  factors  are  considered  in  the  promotion  process  (e.g.,  time  requirements  and 
availability  of  promotion  slots).  In  addition,  promotions  at  higher  ranks  might  be  thought  of  as  a 
“tournament”  in  which  all  the  candidates  are  highly  qualified.  Consequently,  there  might  be  little 
variation  among  candidates  for  promotion  in  terms  of  their  job  performance.  For  example,  all  the 
candidates  might  have  participated  in  a  continuing  education  program.  Consequently,  gross 
measures  of  participation  may  not  show  an  effect  on  promotions.  However,  more  refined 
measures,  such  as  the  number  of  courses  or  semester  hours  taken,  may  distinguish  among 
candidates,  even  when  all  of  them  have  participated  to  some  extent.  In  addition,  measures  that 
indicate  a  long-term  commitment  to  CE,  such  as  completion  of  the  requirements  for  a  degree,42 
may  also  distinguish  among  candidates  for  promotion  at  higher  levels. 

Using  promotions  to  measure  job  performance  introduces  a  second  conceptual  problem 
for  this  part  of  the  evaluation  -  namely,  that  promotion  points  are  given  for  the  successful 
completion  of  specific  ACES  programs.43  Thus,  ACES  program  participation  has  a  direct  effect 
on  promotions  regardless  of  the  effect  of  program  participation  on  actual  job  performance.  Use 
of  a  more  refined  measure  of  participation,  such  as  semester  hours  taken,  and  consideration  of 
degrees  attained  should  make  it  possible  to  identify  effects  of  ACES  participation  in  addition  to 
those  caused  directly  by  the  promotion  points  that  are  earned  from  the  participation. 

One  possible  approach  to  determine  the  effect  of  ACES  participation  above  and  beyond 
the  promotion  points  is  includes  the  following  steps.  First,  determine  the  expected  number  of 


41  Garcia  et  al.  (1998)  analyzed  demotions  as  an  indicator  poor  job  performance.  Demotions,  however,  often  occur 
for  disciplinary  reasons  that  are  not  directly  related  to  a  soldier’s  ability  to  perform  his  or  her  job. 

42  Degree  attainment  is  not  recorded  directly,  either  in  EDMIS  or  through  SOCAD.  It  can  be  inferred  from  a  change 
in  the  educational  credentials  recorded  in  administrative  records. 

43  Formal  education  is  an  important  component  of  a  soldier’s  professional  development.  The  Army  recognizes  a 
soldier’s  educational  attainment  by  awarding  promotion  points,  for  some  ranks,  when  the  soldier  achieves  certain 
scholastic  milestones  such  as  completing  courses  or  completing  an  academic  degree.  For  example,  for  some  ranks, 
promotion  points  are  awarded  for  obtaining  a  high  school  diploma  or  general  education  development  (GED) 
equivalent,  for  completing  any  accredited  post-secondary  test  while  on  active  duty,  and  for  each  semester  hour 
earned  at  an  accredited  trade  school,  college,  or  university.  Furthermore,  enlisted  soldiers  cannot  be  promoted  to  the 
rank  of  Sergeant  or  higher  without  a  high  school  diploma  or  GED.  Data  on  participation  in  the  TA  program  are 
available  in  the  EDMIS.  Promotion  points  are  awarded  for  satisfactory  completion  of  the  DANTES  sponsored 
examination  program,  the  CLEP  general  and  subject  examinations,  and  the  ACT  proficiency  exams.  Also,  see 
footnote  #20. 
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promotion  points  the  soldier  would  receive  based  on  the  number  of  college  credit  hours  taken  or 
degrees  attained.  Second,  using  data  on  ACES  participants,  estimate  a  logistic  regression  model 
of  the  probability  of  promotion.  The  explanatory  variables  in  that  regression  would  include  the 
estimated  number  of  promotion  points  given  for  ACES  participation,  measures  of  ACES 
participation  that  do  not  result  in  promotion  points  (e.g.,  college  credit  hours  above  the 
maximum  used  to  award  promotion  points),  and  other  variables  that  are  expected  to  affect  the 
probability  of  promotions  (e.g.,  MOS  and  time  in  grade).  A  significant  effect  of  ACES 
participation  variables  in  this  model  would  indicate  that  participation  in  ACES  has  a  direct  effect 
on  promotion  in  addition  to  the  effect  that  it  has  on  promotion  points. 

In  addition,  the  evaluation  can  look  at  job  reclassification  as  a  measure  of  how  ACES 
affects  job  performance  in  the  military.  Although  job  reclassification  is  not  a  direct  measure  of 
soldier  performance,  if  a  soldier  is  reclassified  to  an  MOS  that  better  matches  his  or  her  abilities 
or  his  or  her  career  goals  then  a  reclassification  can  affect  military  readiness  via  its  indirect 
effects  on  job  performance  and  career  satisfaction.  Not  all  changes  in  MOS  will  be  counted  as  a 
reclassification  for  purposes  of  this  analysis.  For  example,  changes  in  MOS  that  occur  because 
an  MOS  is  discontinued  or  renamed  will  not  be  counted  as  a  reclassification  in  this  analysis. 
Likewise,  MOS  changes  that  occur  automatically  as  the  result  of  a  promotion  (or  a  demotion) 
will  not  be  counted  as  reclassification  in  this  analysis. 

The  Conceptual  Model 

Many  factors,  including  the  academic  training  and  leadership  courses  available  through 
ACES,  determine  how  well  a  soldier  performs  his  or  her  job.  Conceptually,  one  can  think  of  the 
relationship  between  soldier  performance  and  the  determinants  of  performance  as  a  production 
function: 


performance  =  f(E,  T,  X), 

where  performance  is  determined  in  part  by  a  soldier’s  education  (E),  training  and  experience 
(7),  and  other  factors  ( X )  such  as  personal  characteristics  (e.g.,  aptitude  and  motivation)  and  job 
characteristics  (e.g.,  MOS). 

Performance,  in  this  conceptual  model,  is  considered  the  output  while  education,  training 
and  experience,  and  selected  soldier  characteristics  are  considered  inputs.  An  increase  in  one 
input  into  the  production  function  is  presumed  to  increase  the  output  (i.e.,  performance), 
although  the  marginal  increase  in  performance  from  a  unit  increase  in  the  input  might  not  be 
constant.  For  example,  the  relationship  might  be  one  of  diminishing  returns  where  the  first  unit 
of  the  input  (e.g.,  one  year  of  college)  has  a  larger  effect  on  performance  than  the  second  unit  of 
the  input  (e.g.,  the  second  year  of  college),  and  so  on. 

In  some  cases  the  inputs  might  be  substitutes  and  thus  interchangeable.  In  other  cases  the 
inputs  might  be  complements  in  which  performance  does  not  increase  unless  more  than  one  of 
the  inputs  is  increased.  In  some  cases  the  inputs  might  interact  so  that  increasing,  say,  education 
might  affect  the  relationship  between  training  and  performance.  To  illustrate  these  points, 
consider  the  following  examples.  Education  and  soldier  motivation  are,  to  some  degree, 
substitutes.  A  highly  motivated  but  less  educated  soldier  might  perform  at  the  same  level  as  a 
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less  motivated  but  more  highly  educated  soldier.  In  addition,  these  two  inputs  likely  interact  in 
the  production  function.  That  is,  the  increase  in  performance  for  a  highly  motivated  soldier  who 
participates  in  an  educational  program  might  be  different,  and  presumably  higher,  than  the 
increase  in  performance  for  a  less  motivated  soldier  who  participates  in  the  same  educational 
program. 

Similarly,  in  many  cases  education  and  training  might  be  considered  substitutes. 
Likewise,  education  and  training  could  interact  such  that  education  enhances  the  effect  of 
training  on  performance.  The  direct  effect  of  education  on  job  performance,  and  the  indirect 
effect  of  education  on  performance  via  its  effect  on  training  are  illustrated  in  Figure  2.  These 
interaction  effects  are  incorporated  into  the  conceptual  model  by  modifying  the  production 
function  discussed  above: 


performance  =  f{E(E,T,X),T(E,T,X),X], 

where  E(E,  T,  X)  is  the  direct  effect  of  education  on  job  performance  and  is  a  function  of  the  type 
and  amount  of  education  received,  training  received  prior  to  participation  in  a  continuing 
education  program,  and  soldier  attributes  such  as  motivation  and  intelligence;  and  T(E,  T,  X)  is 
the  indirect  effect  of  education  on  job  performance  via  its  effect  on  trainability.44 


Figure  2.  Relationship  between  education  and  performance. 


44  One  important  question  is  whether  educational  attainment  enhances  trainability,  or  whether  educational  attainment 
simply  is  an  indicator  of  a  person’s  trainability. 
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Hypotheses  Derived  from  the  Conceptual  Model 

From  the  conceptual  model  one  can  derive  testable  hypotheses  regarding  the  effect  of 
various  ACES  programs  on  soldier  job  performance  as  measured  by  promotions  and  timing  of 
promotions.  As  discussed  previously.  Cohorts  3a  and  3b  are  designed  to  estimate  the  impact  of 
ACES  participation  on  promotion  to  E5,  and  Cohorts  4a  and  4b  are  designed  to  analyze 
promotions  to  E6.  Examples  of  testable  hypotheses  to  address  the  questions  posed  in  the 
Introduction  include  the  following. 

•  Participation  in  the  Army’ s  Tuition  Assistance  Program  increases  the  likelihood  of  an 
early  promotion  to  E5  and  E6. 

•  Participation  in  NCO  leadership  development  courses  increases  the  likelihood  of  an  early 
promotion. 

•  The  impact  of  ACES  participation  on  the  likelihood  and  timing  of  promotions  differs  by 
type  of  soldier  (e.g.,  high  school  graduates  verses  non-graduates),  by  military  job 
characteristics  (e.g.,  by  MOS),  and  by  the  type  of  academic  courses  taken  through  the 
ACES  program. 

•  Participation  in  ACES  increases  the  likelihood  of  an  MOS  reclassification. 

Model  Specification  and  Estimation 

Ideally,  the  impact  of  continuing  education  on  job  performance  would  be  measured  using 
a  “fixed-effects”  model  where  information  on  job  performance  was  collected  both  prior  to  and 
after  completing  the  educational  course.  Then,  each  soldier  would  be  his  or  her  own  control 
group.  Unfortunately,  such  an  analysis  is  not  possible  given  the  lack  of  a  job  performance 
measure  that  could  be  assessed  before  and  after  participating  in  the  education  program. 

Consequently,  the  conceptual  model  described  above  should  be  estimated  using  cross- 
sectional  data  with  a  control  for  self-selection  into  the  ACES  program.  Logistic  regression  can  be 
used  to  model  the  impact  of  ACES  participation  on  whether  a  soldier  is  promoted  or  changes 
MOS  during  the  period  of  observation.  Two  approaches  could  be  used  to  determine  whether 
ACES  participation  affects  the  timing  of  promotions.  One  approach  is  to  estimate  a  survival 
model  (similar  to  the  approach  used  to  model  attrition).  The  second  approach  is  to  predict  the 
time  to  promotion  based  on  ACES  participation  using  an  ordinary  least  squares  (OLS)  model. 
Both  approaches  have  similar  data  requirements,  but  survival  analysis  provides  a  better  treatment 
of  censored  data.  Consequently,  it  is  the  preferred  method  for  estimating  the  effect  of  ACES 
participation  on  the  timing  of  promotions.  Table  10  and  Table  1 1  provide  a  list  of  explanatory 
variables  for  the  promotion  and  reclassification  analyses,  respectively.  Explanatory  variables  for 
the  promotion  analysis  were  identified  in  the  literature  review  as  summarized  in  Table  2.  We  are 
not  aware  of  any  comparable  research  regarding  reclassification,  and  so  have  proposed  the  same 
explanatory  variables  to  be  used  for  that  analysis. 

In  addition  to  estimating  logistic  regression  models  to  determine  the  impact  of  ACES 
participation  on  promotions  and  MOS  reclassification,  the  analysis  should  provide  tabulations 
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depicting  the  number  of  soldiers  who  participate  in  various  ACES  programs  and  compare 
promotion  and  reclassifications  rates  for  soldiers  who  do  and  do  not  participate  in  the  various 
programs.  Also,  the  analysis  should  provide  separate  analyses  for  subgroups  of  soldiers — e.g.,  by 
rank,  sex,  race/ethnicity,  and  occupation. 

The  Cost-Benefit  Analysis 

Information  on  the  costs  and  benefits  of  the  ACES  program  to  the  Army  is  vital  to  allow  the 
Army  to  allocate  resources  to  those  activities  that  are  most  beneficial  to  the  service  and  its 
members.  The  preceding  sections  describe  the  process  for  calculating  the  benefits  of  ACES  in 
terms  of  increased  soldier  retention  and  productivity.  This  section  describes  the  process  for 
quantifying  the  costs  and  benefits  of  ACES  in  dollars.  Additional  costs  and  benefits  that  cannot 
be  quantified  in  dollars  are  identified. 

Table  10 

Potential  Explanatory  Variables  for  the  Promotion  Analysis 


Explanatory  Variables 

Data  Source 

Academic  Achievement/Ability  at  Accession 

•  AFQT  percentile 

PEF 

•  Have  GED 

PEF  ^ 

•  High  school  diploma 

PEF 

•  Some  college 

PEF 

•  Changes  in  education  level  since  accession 

PEF 

Demographic  Characteristics 

•  Age45 

PEF 

•  Sex 

PEF 

•  Marital  status 

PEF 

•  Changes  in  marital  status 

PEF 

*  Race/ethnicity 

PEF 

Career 

•  Occupation 

PEF 

•  Changes  in  occupation  since  accession 

PEF 

Education  Program  Participation  Since  Last  Promotion 

•  The  Army  Tuition  Assistance  Program 

EDMIS 

•  The  SOCAD  Program 

SOCAD  Agreement 

Database 

•  MOS  Improvement  Courses 

EDMIS 

•  NCO  Leader  Development  Courses 

EDMIS 

45  Age  at  time  of  promotion  to  E-4  (to  analyze  E-5  promotions)  and  age  at  time  of  promotion  to  E-5  (to  analyze  E-6 
promotions). 
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Table  11 

Potential  Explanatory  Variables  for  the  MOS  Reclassification  Analysis 


Explanatory  Variables 

Data  Source 

Academic  Achievement/  ability  at  Accession 

•  AFQT  percentile 

PEF 

•  Have  GED 

PEF 

•  High  school  diploma 

PEF 

•  Some  college 

PEF 

•  Changes  in  education  level  since  accession 

1 Demographic  Characteristics 

•  Age 

PEF 

•  Sex 

PEF 

•  Marital  status 

PEF 

•  Changes  in  marital  status 

PEF 

•  Race/ethnicity 

PEF 

Career 

•  Occupation 

PEF 

Education  Program  Participation  Since  Last  Change  in 
MOS 

•  The  Army  Tuition  Assistance  Program 

EDMIS 

•  The  SOCAD  Program 

SOCAD  Agreement 

Database 

•  MOS  Improvement  Courses 

EDMIS 

•  NCO  Leader  Development  Courses 

EDMIS 

Measures  of  Cost  Effectiveness 


Soldiers  are  considered  a  human  capital  asset.  The  military  incurs  costs  to  increase  this 
asset,  but  also  derives  benefits  in  the  form  of  higher  levels  of  readiness.  Like  all  assets,  various 
measures  exist  that  allow  the  Army  to  quantify  its  return  on  investment  in  human  capital.  The 
most  appropriate  measure  for  this  analysis  is  the  Net  Present  Value  (NPV). 

The  NPV  is  the  discounted  future  benefits  minus  discounted  costs  and  is  calculated: 


NPV  = 


where  Bt  and  Ct  are,  respectively,  the  benefits  and  costs  to  the  Army  of  ACES  participation  at 
time  “f ’,  and  “T”  is  the  expected  date  of  separation  from  the  Army.  If  the  NPV  is  greater  than 
zero,  then  program  costs  are  more  than  offset  by  program  benefits. 

Discounting  is  important  for  this  evaluation  because  the  cost  of  ACES  participation  and 
the  benefits  derived  might  occur  in  different  time  periods.  For  example,  soldier  participation  in 
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continuing  education  might  reap  immediate  returns  in  terms  of  increased  soldier  productivity,  but 
the  returns  in  terms  of  improved  retention  might  not  be  realized  until  several  years  in  the  future. 
Most  costs  to  the  Army  when  a  soldier  participates  in  an  ACES  program  are  incurred  up  front. 
The  timing  of  costs  and  benefits  is  of  interest  to  the  Army  to  ensure  the  Army  receives  a  return 
on  its  investment.  If  completion  of  the  academic  training  occurs  early  in  the  soldier’s  military 
career,  then  the  military  has  time  to  recoup  the  investment.  If,  on  the  other  hand,  the  training 
occurs  immediately  prior  to  separation  from  the  military,  then  the  military  has  little  time  to 
recoup  the  training  costs. 

Estimating  Costs 

Budget  costs  for  many  of  the  ACES  programs  and  services  are  readily  available.  The  cost 
estimates  used  in  this  evaluation,  however,  should  include  hidden  program  costs  that  might  not 
be  included  in  the  ACES  program  budget— such  as  the  cost  of  health  benefits,  retirement,  and 
leave  for  program  personnel,  and  other  overhead  costs. 

Although  most  ACES  programs  provide  soldiers  with  the  opportunity  to  receive 
continuing  education  during  off-duty  hours,  soldiers  who  participate  in  ACES  might  spend  some 
on-duty  time  participating  in  an  ACES  program.  For  example,  counseling  services  might  be 
available  only  during  working  hours.  The  cost  to  the  Army  of  soldier  on-duty  time  spent 
participating  in  ACES  should  be  included  in  the  cost  estimates. 

Estimating  Benefits 

The  expected  major  benefits  of  ACES  participation  are  higher  retention  and  improved  job 
performance. 

Reduced  soldier  separation  and  replacement  costs.  The  value  of  a  soldier  to  the  Army  is 
not  easily  quantifiable,  but  the  separation  and  replacement  costs  resulting  from  attrition  and 
decisions  not  to  reenlist  can  be  quantified.  Separation  costs  include  expenses  incurred  to  process 
the  departing  soldier  out  of  the  Army,  as  well  as  pecuniary  costs  incurred  by  the  Army  for 
benefits  promised  to  the  soldier  following  separation.  Replacement  costs  consist  of  recruiting 
and  training  expenses. 

Table  12  identifies  specific  separation  and  replacement  expenses  incurred  by  the  Army 
when  soldiers  separate.  Separation  and  replacement  costs  will  vary  substantially  by  soldier 
depending  on  where  soldiers  are  located,  job  type  and  years  in  the  service. 

The  net  cost  to  the  Army  of  losing  a  soldier  for  failure  to  reenlist  should  include  all  the 
costs  listed  above  minus  costs  the  Army  would  have  incurred  if  the  soldier  had  re-enlisted— such 
as  selective  reenlistment  bonuses, 

Garcia  et  al.  (1998)  estimated  the  replacement  cost  to  the  Navy  for  each  enlisted  sailor 
who  selects  not  to  reenlist  is  approximately  $24,301  in  FY  98  dollars.  This  estimate,  though, 
appears  not  to  take  into  account  attrition  during  various  stages  of  the  first  enlistment.  For 
example,  if  one  third  of  accessions  separate  from  the  military  prior  to  the  end  of  the  first 
enlistment,  then  the  military  must  recruit  approximately  150  persons  to  replace  100  persons  who 
choose  not  to  reenlist  at  the  first  enlistment  decision.  The  military  will  incur  recruiting  costs  and 


varying  levels  of  training  costs  for  all  of  these  150  accessions.  Likewise,  this  estimate  from 
Garcia  et  al.  does  not  appear  to  include  separation  costs. 

Table  12 

Soldier  Separation  and  Replacement  Costs 

_ _ Separation  Costs _ 

Outprocessing 

•  Productivity  loss  when  soldier  is  not  functioning  with  unit 

•  Administrative  costs 
Relocation 

•  Soldier’s  household  relocated 

•  Administrative  costs _ 

Post-separation  benefits 

•  G.I.  Bill _ 

_ Recruiting  Costs 

Recruiting  costs 

•  Advertising 

•  Recruiters 

•  Testing  (e.g.,  MET  &  MEPS  centers) 


•  Administrative  costs 


Training  Costs 

Training  costs 

• 

BASIC 

• 

AIT 

• 

Training  exercises 

• 

Other  training  programs 

Some  costs  and  benefits  of  separation  are  subtler  and  more  difficult  to  measure.  For 
example,  some  researchers  have  suggested  that  turnover  has  a  demoralizing  effect  and  thus 
breeds  additional  turnover.  On  the  other  hand,  turnover  might  create  promotion  opportunities  for 
those  who  remain,  which  has  a  positive  effect  on  retention  for  those  who  remain.  Also,  a  portion 
of  soldiers  who  leave  active  duty  will  enter  the  Army  Reserves  or  National  Guard,  which  offsets 
part  of  the  total  loss  to  the  Army  when  a  soldier  separates. 

Griffeth  and  Horn  (1995)  suggest  that  exit  costs  might  be  overstated  because  they  ignore 
the  identity  of  employees  who  leave.  For  example,  if  poor-performing  soldiers  are  more  likely  to 
leave  than  good  performers  then,  on  average,  the  replacement  soldier  will  be  of  higher  quality 
than  the  separating  soldier.  It  might  be  in  the  best  interest  of  the  Army  if  low-quality  soldiers 
leave  after  their  first  enlistment.46 


46  Dalton,  Todor,  and  Krackhardt  (1982)  introduced  a  taxonomy  to  classify  turnover  and  identify  turnover  as  “bad” 
if  high  performers  leave  or  low  performers  stay,  and  “good”  if  high  performers  stay  and  low  performers  leave.  This 
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Higher  productivity.  As  described  previously,  the  unavailability  of  precise  measures  of 
soldier  productivity  has  led  many  researchers  to  use  promotions  as  an  indicator  of  improved 
soldier  performance.  In  the  civilian  literature,  however,  many  studies  assume  that  an  employee’s 
wages  equal  his  or  her  marginal  productivity  and  thus  use  changes  in  wages  as  a  proxy  for 
changes  in  productivity. 

In  the  military,  this  assumption  that  compensation  equals  marginal  productivity  is  not 
valid.  The  Army  invests  significant  resources  to  train  soldiers.  Increased  training  increases  the 
value  of  soldiers  to  the  Army  with  only  a  small  and  indirect  effect  on  compensation.  Thus, 
changes  in  pay  likely  would  underestimate  change  in  productivity.  In  terms  of  benefits  to  the 
Army,  if  changes  in  performance  were  perfectly  offset  by  changes  in  compensation,  then  the  net 
benefit  in  dollars  of  improved  performance  to  the  Army  would  be  zero.  In  summary,  if  the 
evaluation  finds  that  ACES  participation  increases  promotions  (or  reduced  the  time  to 
promotion),  then  the  evaluation  could  consider  using  the  increase  in  basic  pay  to  quantify  the 
value  of  increased  performance.  This  measure  has  numerous  limitations,  though. 

Summary 

The  purpose  of  this  evaluation  plan  is  to  help  ensure  a  thorough  and  rigorous  evaluation 
of  the  impact  of  ACES  on  soldier  retention  and  job  performance.  The  plan  first  identifies 
questions  that  should  be  addressed  in  the  evaluation  and  testable  hypotheses  to  address  these 
questions.  Then,  the  plan  identifies  data  sources  and  modeling  techniques  to  test  these 
hypotheses. 

In  addition,  the  evaluation  plan  identifies  data  and  methodological  issues  that  the 
evaluation  must  address  to  ensure  accurate  findings.  The  most  serious  issue  is  that  soldiers  select 
to  participate  in  ACES.  Many  of  the  factors  that  are  hypothesized  to  increase  the  propensity  to 
participate  in  ACES  are  also  hypothesized  to  affect  soldier  retention  and  job  performance. 
Failure  to  control  for  selection  bias  will  likely  result  in  overestimation  of  the  impact  of  ACES  on 
Army  readiness.  Several  approaches  and  statistical  techniques  are  proposed  to  mitigate  the 
adverse  effects  of  selection  bias  on  the  program  impact  estimate. 

Two  types  of  data  sources  are  discussed.  The  first  is  the  1999  Survey  of  Active  Duty 
Personnel.  This  survey  is  a  rich  source  of  data  for  an  analysis  of  reenlistment  intentions  and 
contains  variables  that  indicate  whether  a  soldier  participated  in  certain  ACES  programs  during 
the  previous  12  months.  The  second  source  of  data  encompasses  several  administrative 
databases.  Data  on  ACES  participation  from  EDMIS  and  SOCAD  databases  can  be  merged  with 
data  on  promotion  and  retention  outcomes  and  soldier  characteristics  available  in  databases  such 
as  the  PEF  and  Project  First-Term  Database.  The  administrative  databases  should  provide  more 
precise  measures  of  ACES  participation  and  soldier  outcomes  than  will  the  S  ADP.  The  SADP, 
however,  contains  data  on  soldier  intentions  and  satisfaction  with  various  facets  of  the  military 
that  are  not  available  in  administrative  databases.  Furthermore,  preparing  an  analysis  database 
from  SADP  will  require  substantially  fewer  resources  than  preparing  an  analysis  database  from 


concept  relies  on  the  assumption  that  employers  will  replace  employees  who  leave  with  employees  of  at  least 
average  performance.  Consequently,  low  performers  who  leave  are  replaced  with  better  performer,  on  average. 
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administrative  files.  In  addition,  the  SADP  can  potentially  be  linked  to  the  administrative  data  to 
provide  an  even  richer  database  with  which  to  analyze  the  impact  of  ACES  on  outcome  measures 
and  control  for  soldiers’  satisfaction  with  various  aspects  of  military  life.  For  the  reasons 
discussed  above,  both  data  sources  are  important  to  this  evaluation.  The  merged 
SADP/administrative  data  file  will  be  analyzed  provided  the  sample  size  from  the  merged  files  is 
sufficient  to  conduct  meaningful  analyses. 

Four  analyses  are  proposed  in  the  evaluation  plan:  (a)  a  reenlistment  analysis,  (b)  an 
attrition  analysis,  (c)  a  reclassification  analysis  and  (d)  a  promotion  analysis.  These  analyses  can 
be  completed  using  conventional  modeling  techniques  such  as  OLS  regression,  logistic 
regression,  probit  analysis,  and  survival  analysis.  Logistic  regression  and  probit  analysis  can  be 
used  to  model  whether  soldiers  reenlist,  and  whether  they  complete  their  first  enlistment.  OLS 
regression  can  be  used  to  predict  the  time  to  promotion.  Survival  analysis  can  be  used  to  model 
the  timing  of  both  attrition  and  promotions.  The  retention  analysis  should  focus  on  soldiers  in 
their  first  and  second  enlistment,  while  the  attrition  study  should  focus  on  soldiers  in  their  first 
enlistment.  Because  most  attrition  occurs  early  in  the  first  enlistment  before  a  soldier  has  the 
opportunity  to  participate  in  ACES  (or  even  become  aware  of  ACES),  the  impact  of  ACES  on 
attrition,  if  any,  is  likely  to  be  small.  The  promotion  analysis  should  focus  on  promotions  to  E-5 
and  to  E-6. 

The  evaluation  plan  discusses  ways  to  quantify  the  costs  and  benefits  of  ACES  using 
conventional  cost-benefit  measures.  This  information  can  be  used  to  compare  various  ACES 
programs  in  terms  of  their  value  added  to  the  Army,  as  well  as  how  ACES  programs  compare  to 
other  Army  programs  in  its  contribution  to  Army  readiness. 
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DATABASE  DEVELOPMENT  PLAN 


Database  Plan  Objectives 

The  data  requirements  of  an  evaluation  study  can  be  complex,  drawing  from  numerous 
and  diverse  data  sources.  This  database  development  plan  has  a  single  objective:  to  ensure  a 
sound  and  successful  study  by  meeting  all  the  data  needs  of  the  evaluation  in  the  most  efficient 
manner  possible.  This  objective  will  be  accomplished  in  three  ways.  First,  the  database  plan  will 
be  based  on  a  thorough  assessment  of  the  data  requirements  of  the  evaluation  study.  Therefore, 
the  database  plan  will  rely  on  the  evaluation  plan  for  such  critical  information  as  research  design, 
statistical  approach,  identification  of  populations  of  interest,  and  data  elements  required.  Both  the 
evaluation  and  database  plans  draw  heavily  from  one  another.  While  it  is  clear  that  the  database 
plan  should  be  defined  by  the  goals  and  requirements  of  the  evaluation  plan,  it  is  also  true  that 
the  latter  cannot  put  forth  reasonable  guidelines  without  considering  the  limitations  and 
availability  of  the  former. 

Second,  this  database  development  plan  will  review  all  relevant  data  sources,  providing  a 
discussion  of  the  “goodness  of  fit”  between  the  attributes  of  data  from  a  number  of  sources  and 
the  requirements  of  the  evaluation.  The  discussion  will  conclude  with  recommendations 
identifying  those  data  sources  that  provide  the  richest,  most  efficient,  and  relevant  data  for  use  in 
the  evaluation  study.  Third,  the  database  plan  will  provide  detailed  guidelines  for  data 
procurement  and  the  creation  of  an  analytic  database  that  will  serve  the  complete  data  needs  of 
the  phase  two  ACES  evaluation. 

Evaluation  Plan  Data  Requirements 


Population  Characteristics 

Different  ACES  programs  are  designed  to  benefit  soldiers  at  different  points  in  their 
careers.  For  example,  while  participants  in  the  FAST  program  tend  to  be  newer  accessions,  NCO 
Leadership  courses  are  designed  solely  for  current  or  soon-to-be-NCOs.  In  addition  to  the  fact 
that  some  ACES  programs  are  relevant  only  at  specific  points  in  a  soldier’s  career,  evidence  of 
the  evaluation  study’s  criterion  measures  is  also  often  apparent  only  at  particular  points  in  time. 
For  example,  attrition  is  mostly  a  phenomenon  of  new  accessions:  nearly  half  of  first-term 
attrition  occurs  within  the  first  12  months  of  service  (Laurence,  Naughton,  &  Harris,  1995). 
Obviously,  the  earliest  evidence  of  reenlistment  can  occur  only  at  or  near  the  conclusion  of  the 
first  term  of  service,  and,  thereafter,  at  the  end  of  each  subsequent  term.  Promotions  from  E-2 
through  E-4  are  generally  automatic,  based  on  time-in-grade  and,  therefore,  not  of  great  analytic 
interest  to  the  evaluation.  Promotions  to  higher  grades,  however,  which  are  typically  based  on 
performance  and  education,  generally  do  not  occur  before  completion  of  the  first  term  of  service. 

In  addition  to  these  temporal  constraints,  there  is  the  added  limitation  of  the  availability 
°f  ACES  participation  data.  EDMIS  is  the  Army’s  operational  database  containing  information 
on  ACES  participation.  As  stated  in  the  previous  section,  EDMIS  was  installed  in  a  subset  of 
CONUS  installations  as  early  as  1995,  but  nearly  all  Army  bases,  including  OCONUS  sites,  had 
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EDMIS  installed  by  January  1999.47  The  lack  of  complete  EDMIS  data  prior  to  January  1999  is  a 
serious  data  constraint  that  will  be  discussed  in  detail  later. 

Analysis  Cohort  Definitions 

In  defining  its  analytic  cohorts,  the  evaluation  plan  takes  into  account  a  number  of 
temporal  issues,  some  of  which  have  already  been  mentioned.  First,  as  discussed  above,  evidence 
of  the  criterion  measures  and  the  relevance  of  various  ACES  programs  for  particular  soldiers  are 
both  related  to  the  specific  stage  in  a  soldier’s  career  and  substantially  affect  cohort  definitions. 
Second,  EDMIS  data  availability  sharply  limits  the  time  period  that  education  behavior  in  a 
cohort  of  soldiers  can  be  observed.  Lastly,  the  expected  starting  date  of  the  evaluation  study  and, 
more  importantly,  its  expected  duration  will  influence  cohort  definitions.  The  evaluation  plan 
assumes  that  the  phase  two  evaluation  study  will  commence  in  late  FY  2001  and  will  be 
completed  by  the  end  of  June  2003.  While  sufficient  time  must  be  allocated  for  database 
preparation  and  analysis,  data  must  be  collected  at  the  latest  possible  date  to  allow  for  the 
inclusion  of  as  much  automated  education  participation  data  as  possible  in  EDMIS.  Assuming 
that  three  months  is  required  to  collect  evaluation  data  and  a  little  over  a  year  to  prepare  the 
database,  analyze  the  data,  and  write  the  evaluation  report(s),  then  the  project  data  will  likely  be 
collected  starting  in  January  2002  to  capture  data  through  December  2001.  Following  this 
schedule,  the  evaluation  study  data  will  reflect,  at  best,  three  years  of  soldier  education  behavior 
(January  1999  through  December  2001).48 

The  evaluation  plan  identifies  five  main  Regular  Army  cohorts  to  be  used  in  the 
evaluation.  The  first  analysis  cohort  is  defined  as  non-prior  service  FY  1999  enlisted  accessions 
and  will  form  the  population  used  to  analyze  early  attrition  and  reclassification.49  Second, 
reenlistment  analyses  will  be  conducted  on  all  enlisted  soldiers  with  an  ETS  date  in  calendar  year 
2001. 50  Because  this  cohort  includes  soldiers  in  all  enlistment  terms  terminating  in  2001,  the 
relationship  between  ACES  participation  and  first  and  all  subsequent  reenlistments  can  be 
assessed  from  this  cohort.  Third,  the  relationship  between  ACES  participation  and  promotion  to 


All  but  8  facilities  had  EDMIS  installed  by  December  1998.  EDMIS  was  not  installed  in  Europe,  Japan,  and 
Korea  until  September  1998,  October  1998,  and  December  1998,  respectively. 

48  It  is  important  to  note  that  facilities  require  time  to  learn  the  EDMIS  system  and  set  up  the  database  correctly, 
although  how  much  time  is  required  is  unclear.  Therefore,  while  three  years  of  complete  EDMIS  data  will  exist’ as  of 
December  2001,  some  of  these  data  may  be  unreliable,  particularly  in  sites  that  began  using  EDMIS  later  in  the 
EDMIS  installation  period  of  1995-1998. 

49For  the  1999  cohort,  soldier  careers  will  be  observed  for  a  maximum  period  of  36  months  and  a  minimum  of  24 
months,  for  January  and  December  1999  accessions,  respectively.  Except  for  two-  and  some  three-year  enlistments, 
the  criteria  are  not  “first-term”  but  rather  “early”  attrition  and  reclassification  because  the  data  do  not  allow  the 
observation  of  soldier  careers  across  the  typical  four-year  term. 

50  Reenlistment  can  occur  up  to  8  months  prior  to  the  ETS  date.  Once  reenlisted,  a  soldier’s  ETS  date  changes. 
Therefore,  to  identify  only  soldiers  who  had  a  2001  ETS  date  in  2001  data  would  miss  those  soldiers  who  had  a 
2001  ETS,  but  reenlisted  prior  to  2001.  To  circumvent  this  problem,  cohort  members  will  be  identified  based  on 
their  ETS  date  9  months  prior  to  the  beginning  date  of  the  period.  In  this  case,  cohort  members  would  be  identified 
from  April  2000  data. 
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E-5  will  be  studied  using  the  cohort  of  enlisted  soldiers  who  were  promoted  to  E-4  during 
calendar  year  1998.  Fourth,  enlisted  soldiers  promoted  to  E-5  in  calendar  year  1998  will  form  the 
cohort  used  to  analyze  promotion  to  E-6.  The  cohorts  used  for  both  promotion  analyses  will 
consist  of  soldiers  from  different  accession  years  and  with  varying  number  of  enlistment  terms. 
Fifth,  the  relationship  between  education  participation  and  intention  to  reenlist  will  be  analyzed 
on  a  cohort  of  survey  respondents  to  items  on  attitudes  on  military  life,  including  intention  to 
reenlist  and  self-reported  education  participation. 

Finally,  the  evaluation  plan  also  suggests  intersecting  the  survey  cohort  with  the 
reenlistment  and  promotion  cohorts51  to  produce  additional  cohorts  of  interest.  The  feasibility  of 
this  approach  will  depend  in  large  part  on  the  number  of  cases  in  the  resultant  cohorts.  Table  13 
presents  a  summary  of  these  analysis  cohorts,  with  an  indication  of  criterion  measures  that  they 
are  designed  to  study. 


Table  13 

Summary  of  Cohorts  by  Criterion  Measures 


I-'  Cohort  Definitions  Criterion  Measure 

Main  Cohorts: 

FY  1999  enlisted  accessions 

Early  attrition,  early 
reclassification 

Enlisted  soldiers  with  ETS  in  calendar  year  2001 

Reenlistment  (1st  term  and 
beyond) 

Enlisted  soldiers  promoted  to  E-4  in  calendar  year 
1998 

Incidence  and  timing  of 
promotion  to  E-5,  reclassification 

Enlisted  soldiers  promoted  to  E-5  in  calendar  year 
1998 

Incidence  and  timing  of 
promotion  to  E-6,  reclassification 

Survey  respondents 

Intention  to  reenlist 

Intersected  Cohorts: 

Enlisted  soldiers  with  a  calendar  year  2001  ETS 
date  who  are  also  survey  respondents 

Reenlistment  (1st  term  and 
beyond) 

Enlisted  soldiers  promoted  to  E-4  in  calendar  year 
1998  who  are  also  survey  respondents 

Incidence  and  timing  of 
promotion  to  E-5,  reclassification 

Enlisted  soldiers  promoted  to  E-5  in  calendar  year 
1998  who  are  also  survey  respondents 

Incidence  and  timing  of 
promotion  to  E-6,  reclassification 

Data  Elements 

The  evaluation  plan  calls  for  a  number  of  explanatory  variables  to  be  used  in  the  attrition, 
reenlistment,  reclassification,  and  promotion  analyses.  Individual  soldier-level  variables  fall  into 
two  groups:  (a)  variables  that  describe  the  demographic  and  military  job  characteristics  of  a 


31  The  early  attrition/reclassification  cohort  recommended  for  use  later  in  this  report  is  itself  rich  with  attitudinal 
measures.  Therefore,  there  would  be  minimal  benefit  in  intersecting  these  cohorts. 
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soldier  over  a  defined  period  of  time,  and  (b)  those  that  capture  soldier  attitudes,  perceptions,  and 
intentions  at  a  single  point  in  time.  Explanatory  variables  that  describe  the  demographic  and 
military  job  characteristics  of  soldiers  are  either  static  or  fluid  in  nature.  Static  variables,  such  as 
gender  and  race/ethnicity,  are  attributes  that  are  not  expected  to  change  over  the  observation 
period  of  the  study.  Fluid  attributes  are  those  that  can  change  over  time.  For  example,  MOS  and 
CONUS/OCONUS  assignments  are  fluid,  time-varying  attributes  because  they  can  (and  often 
do)  change  over  the  evaluation  study’s  observation  period. 

An  observation  period  is  the  period  of  time  during  which  the  evaluation  study  examines 
data  that  describe  a  soldier’s  career  and  behavior.  Observation  periods  differ  by  cohort.  Table  14 
presents  the  observation  period  beginning  and  ending  points  for  the  four  main  cohorts  that  utilize 
soldier-level  demographic  and  military  job/career  data. 

Table  14 

Summary  of  Cohort  Observation  Periods 


Cohort 

Analysis 

3  Obsen 

ation  Period  ' 

'  '  3  /  >■'  x  f*.L 

gi  fjjf  '  Begin-  1||| 

,,  End 

1999  Accessions 

Attrition, 

Reclassification 

Enlistment  date  in  1999 

•  Date  of  Separation 

•  Date  of  Reenlistment 

•  Data  Cut-off  Date 

Soldiers  with 

ETS  in  2001 

Reenlistment 

Enlistment/Reenlistment 

Date 

•  Date  of  Separation 

•  Date  of  Reenlistment 

•  Data  Cut-off  Date 

Soldiers 

promoted  to  E-4 
in  199S 

Promotion  to  E-5, 
Reclassification 

Date  in  1998  Promoted  to 
E-4 

*  Date  Promoted  to  E-5 

*  Separation 

*  Data  Cut-off  Date 

Soldiers 

promoted  to  E-5 
in  1998 

Promotion  to  E-6, 
Reclassification 

Date  in  1998  Promoted  to 
E-5 

*  Date  Promoted  to  E-6 

*  Separation 

*  Data  Cut-off  Date 

It  is  important  to  note  that  the  end  of  the  observation  period  is  signaled  by  different 
events  for  different  soldiers.  For  example,  one  of  three  events  must  occur  to  each  soldier  to 
punctuate  the  end  of  the  1999  accession  cohort  observation  period:  the  servicemember 
(a)  separates,  (b)  reenlists,52  or  (c)  is  still  on  active  duty  at  the  last  point  in  time  for  which  data 
are  available. 

Given  that  the  goal  of  the  evaluation  is  to  assess  the  effect  of  education  participation  on 
performance  and  retention,  it  is  clear  that  great  care  must  be  taken  to  measure  explanatory 
variables  at  points  in  time  prior  to  the  occurrence  of  the  criterion  measure  being  analyzed.  But  at 
what  point(s)  in  time  should  the  evaluation  capture  time-varying  variables?  Unless  variables 
change  at  predictable  points  during  the  observation  period,  choosing  a  particular  point  in  time  to 
capture  a  fluid  variable  can  be  likened  to  trying  to  hit  a  moving  target:  sometimes  you  can  hit  the 


Reenlistments  in  the  1999  cohort  can  be  observed  for  those  with  an  initial  2-year  enlistment  term  and  for  some 
servicemembers  with  an  initial  3-year  term. 
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target,  but  often  you  don’t.  For  example,  because  soldiers  can  change  MOS  at  any  time,  the 
timing  of  an  MOS  change  cannot  be  predicted.  If  MOS  is  measured  early  in  the  promotion 
observation  period,  then  subsequent  changes  to  MOS  -  which  may  be  highly  related  to 
promotion  -  will  never  be  captured.  Similarly,  looking  late  in  the  observation  period  and 
measuring  only  the  “changed”  MOS  can  also  lead  to  lost  information  -  most  notably,  the  effect 
of  the  original  MOS  on  the  criterion  measure.  To  allow  the  most  precision  in  building  analytic 
models  and  to  provide  the  evaluation  analysts  the  most  flexibility  in  building  those  models,  data 
must  be  captured  as  often  as  possible  during  the  observation  period.  In  other  words,  longitudinal 
data  must  be  collected  to  fully  describe  soldier  demographic  and  military  job  characteristics. 

Table  15  presents  a  list  of  the  explanatory  variables  required  by  the  evaluation  plan  for 
the  attrition,  reenlistment,  reclassification,  and  promotion  analyses.  Time-varying  soldier-level 
demographic  and  military  variables  are  indicated  on  Table  3  with  an  asterisk.  Although 
attitudinal  and  self-reported  education  can  change  over  the  life  of  a  soldier,  they  can  be  captured 
at  a  single  point  in  time  and  are  therefore  not  considered  time- varying  variables  in  Table  15. 

Summary  of  Data  Requirements 

The  evaluation  plan  calls  for  three  types  of  information  on  individual  soldiers:  (a)  data  on 
ACES  participation;  (b)  administrative  data,  including  personal  demographics  and  military  job 
characteristics;  and  (c)  self-reported  education  participation  and  attitudinal  (perceptions  and 
intentions)  measures.  In  addition,  the  evaluation  plan  uses  external,  economic  measures  (e.g., 
unemployment  rates).  Therefore,  the  most  fundamental  requirement  of  a  data  plan  for  this 
evaluation  study  is  that  it  result  in  a  database  that  contains  the  data  elements  listed  in  Table  1553 
on  the  cohorts  specified  by  the  evaluation  plan  for  the  relevant  time  periods.54 

A  further  requirement  is  that  the  data  source(s)  selected  must  be  the  most  cost-  and  labor- 
efficient  of  its  genre,  all  other  factors  being  equal.  In  addition,  the  evaluation  plan  requires 
complete  ACES  participation  data,  regardless  of  soldier  location  and  the  frequency  of  changes  in 
soldier  assignments.  Complete  data  are  important  because  the  variation  in  education  participation 
between  soldiers  should  be  attributable  to  soldier  behavior  rather  than  differences  in  electronic 
record  keeping  (or  the  lack  of  it)  at  various  installations.  It  is  for  this  reason  that  EDMIS  data  are 
deemed  complete  beginning  in  January  1999. 


Additional  variables  to  those  listed  in  Table  15  will  be  requested  for  data  exploration  or  for  use  in  the  construction 
of  other  variables. 

54  Temporal  relevance  dictates  that  the  criterion  measures  post-date  education  participation  and  other  explanatory 
measures. 


Table  15 

List  of  Explanatory  Variables  for  the  Attrition,  Reenlistment  and 
Promotion  Analyses 


> 

Demographic  Characteristics 

♦ 

Date  of  Birth 

♦ 

Gender 

♦ 

Race/Ethnicity 

♦ 

Marital  Status  * 

♦ 

Military  Spouse  * 

♦ 

Number  of  Dependents  * 

♦ 

Education  Level  * 

♦ 

State  of  Legal  Residence  * 

> 

Aptitude  At  Accession 

♦ 

AFQT  Score/Percentile 

♦ 

Education  Level 

> 

Enlistment/Reenlistment  Contract 

♦ 

Term  of  Enlistment/Reenlistment  * 

♦ 

Army  College  Fund  (ACF)  “Kickers” 

♦ 

Montgomery  G,  I.  Bill  Eligibility 

♦ 

Date  of  accession 

♦ 

Date  of  reenlistment  * 

> 

Military  Job  Characteristics 

♦ 

MOS  * 

♦ 

Major  Command  * 

♦ 

Unit  Identification  Code  (UIC)  * 

♦ 

Pay  grade  * 

♦ 

Date  of  separation 

♦ 

Date  of  pay  grade  (promotion)  * 

♦ 

Selective  reenlistment  bonus  (SRB)  multiple 

> 

Satisfaction  with  Military  Life  ! 

♦ 

Satisfaction  with  PERSTEMPO 

♦ 

Intention  to  Reenlist 

♦ 

Other  Satisfaction  Measures 

> 

Career  Attitudes 

♦ 

Career  intentions  at  accession  and  end-of-training 

♦ 

Education  aspirations  at  accession  and  end-of-training 

♦ 

Expectations  regarding  timing  of  next  promotion 

♦ 

Military  versus  civilian  pay  and  benefits 

♦ 

Other  altitudinal  measures 

> 

Access  to  ACES  I 

♦ 

OCONUS/CONUS  *  -1 

♦ 

Installation  has  an  education/leaming  center  * 

♦ 

Member  Works  Extra  Job 

> 

ACES  Participation  1 

♦ 

Self-Reported  Continuing  Education 

♦ 

TA  for  undergraduate  courses  * 

♦ 

Courses  in  SOCAD  schools  * 

♦ 

2/4  year  SOCAD  Agreements  * 

♦ 

FAST  Program  * 

♦ 

MOS  Improvement  Courses  * 

♦ 

NCO  Leader  Development  Courses  * 

> 

Economic  Factors  i 

♦ 

Unemployment  rate  (based  on  Bureau  of  Labor  Statistics  data)  * 

An  asterisk  indicates  fluid,  time-varying,  variables. 
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A  common  unique  person  identifier,  such  as  Social  Security  Number  (SSN),  must  be 
present  in  all  evaluation  data,  so  that  information  from  the  different  data  sources  can  be  merged. 
Finally,  in  order  to  capture  the  full  effect  of  variables  that  change  over  time,  data  sources  must 
provide  longitudinal  data  on  time-varying  soldier  demographic  (including  educational)  and 
military  job  characteristics.  A  summary  of  these  requirements  is  provided  below: 

•  the  combined  data  sources  must  provide  all  the  variables  required  by  the  evaluation  plan; 

•  the  data  source(s)  must  contain  data  on  the  populations  specified  in  the  evaluation  plan 
during  the  appropriate  time  periods; 

•  the  data  source(s)  selected  must  be  the  most  cost-  and  labor-efficient  altemative(s)  of  its 
genre,  all  other  factors  being  equal; 

•  the  data  source(s)  for  objective  education  data  must  result  in  complete  education 
participation  data  for  all  soldiers; 

•  all  data  sources  must  contain  a  common  link  variable  (such  as  SSN);  and 

•  all  time- varying  data  of  interest  must  be  available  longitudinally. 

Data  Sources 

A  discussion  of  data  sources  for  the  types  of  data  of  interest  to  this  evaluation  study  - 
education  participation,  administrative,  self-reported  education  and  attitudinal,  and  economic  -  is 
provided  below. 

A  CES  Participation  Data 

There  are  two  sources  of  electronic  education  participation  data  describing  the  programs 
that  are  included  in  the  evaluation  plan:  EDMIS  and  the  SOCAD  Student  Agreement  databases. 

EDMIS.  ACES  participation  data  are  stored  electronically  as  an  Oracle  relational 
database55  called  EDMIS.56  Since  EDMIS  is  designed  for  distributed  rather  than  centralized 
computing,  the  database  is  populated  and  maintained  at  installation  education/leaming  centers.57 
The  structure  and  format  of  EDMIS  databases  are  the  same  regardless  of  the  installation  site. 
Beginning  in  1995,  six  CONUS  facilities  had  EDMIS  software  installed  for  pilot  testing.  By 


55  Oracle  is  a  corporation  that  produces  a  number  of  software  products.  A  relational  database  is  a  database 
management  system  that  stores  data  in  tables  and  links  tables  through  the  use  of  key  variables.  The  Oracle  system, 
one  of  a  number  of  relational  database  management  systems  available,  makes  it  very  easy  and  efficient  to  link  data 
from  multiple  tables. 

56  The  Directorate  of  Information  Systems,  Communications  Command  Control  and  Computers  provides  funding 
and  staffing  of  all  Information  System  Modules  (ISMs)  of  which  EDMIS  is  a  part. 

57  EDMIS,  like  all  other  ISMs,  is  housed  on  installation  servers,  which  may  be  tapped  by  more  than  one  facility.  For 
example,  one  server  is  used  by  all  European  Army  bases. 
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1997,  all  CONUS  installations  with  an  education/learning  Center  were  using  EDMIS.  Nearly  all 
sites  had  EDMIS  installed  by  the  end  of  calendar  year  1998.58 

The  ACES  participation  history  of  a  soldier  is  recorded  in  the  EDMIS  database  of  the 
facility  at  which  the  soldier  is  stationed  at  the  time  the  participation  occurs.  When  a  soldier  is 
assigned  to  a  new  station,  information  on  educational  activity  occurring  from  that  point  in  time 
until  the  next  assignment  is  stored  in  the  new  station’s  EDMIS  database.  As  stated  previously, 
there  is  no  centralized  repository  of  EDMIS  data.  In  order  to  obtain  complete  ACES  participation 
data  for  a  soldier,  the  EDMIS  data  from  each  facility  in  which  the  soldier  was  located  must  be 
retrieved.  ACES  participation  data  prior  to  January  1999  are  incomplete  for  soldiers  who  moved 
from  an  EDMIS -installed  facility  to  a  non-EDMIS  installed  station.  Even  if  the  evaluation 
focused  only  on  those  soldiers  who  moved  between  EDMIS-installed  facilities  prior  to  1999,  the 
results  may  be  tainted  by  factors  related  to  the  selection  of  sites  for  EDMIS  installation,  such  as 
MOS  or  unit  type.  This  non-random  sample  would  likely  over-represent  certain  types  of  soldiers, 
such  as  those  assigned  to  CONUS  facilities.  Therefore,  in  order  to  be  assured  that  educational 
participation  data  are  complete,  data  must  be  culled  from  all  EDMIS-installed  sites  from  January 
1999  forward.59 

Although  planned  for  the  future,  ACES  data  have  not  been  archived  to  date.  Although 
EDMIS  data  can  be  purged  by  the  installation  based  on  business  necessity,  it  appears  that  no 
EDMIS  data  have  been  purged  to  date.  Therefore,  data  are  available  on-line  since  the  date  of 
EDMIS  installation  at  each  facility.60  Since  EDMIS  is  a  relational  database,  variables  are  stored 
in  tables  that  can  be  merged  with  one  another  by  unique  link  (or  key)  variable(s).  In  addition, 
EDMIS  is  linked  to  the  site  specific  Installation  Level  Integrated  Database  (ILIDB)  which 
contains  data  from  the  installations’  personnel  database.61  The  EDMIS  Installation  Support 
Module  Database  Design  Description  provides  a  list  of  data  elements  contained  within  EDMIS 
tables.  Currently,  there  is  no  EDMIS  codebook  containing  detailed  variable  descriptions  and 
value  definitions.  A  number  of  EDMIS  tables  or  variables  within  tables  are  not  consistently  used 
at  installations  and,  therefore,  have  been  deemed  unreliable  by  EDMIS  experts  at  PERSCOM.62 


58  Ten  sites  had  EDMIS  installed  in  calendar  year  1998;  of  these,  four  OCONUS  sites  (Europe;  Camp  Zama,  Japan; 
Okinawa;  and  Korea)  began  using  EDMIS  in  the  1st  quarter  of  FY  1999.  EDMIS  was  installed  in  Ft.  Buchanan  as 
recently  as  September  1999.  As  of  1  September  2000,  there  are  seven  sites  without  EDMIS:  West  Point;  Dugway 
Proving  Ground;  Charles  Melvin  Price;  Selfridge;  Yuma  Proving  Ground;  Ft  Hamilton;  and  Ft.  Dix. 

59  Although  the  main  analyses  will  be  conducted  on  ACES  participants  as  of  January  1999,  the  evaluation  plan  calls 
for  collecting  EDMIS  data  starting  in  January  1998  -  the  earliest  cohort  date  -  so  that  validation  analyses  may  be 
conducted. 

60  Hard  copy  records  maintained  prior  to  EDMIS  installation  were  input  into  EDMIS  at  some  sites. 

61  The  Standard  Installation/Division  Personnel  System  3  (SIDPERS3)  is  the  Army’s  operational  personnel  database 
installed  and  maintained  at  the  installation-level.  It  is  linked  to  the  ILIDB  and  then  to  EDMIS. 

62  For  example,  all  EDMIS  tables  pertaining  to  the  Army  Personnel  Testing  Program  (APTP)  are  inconsistently 
populated  at  the  sites.  The  SOCAD  tables  (SOCAD,  SOCAD_OTH,  and  SOCREF)  have  also  been  deemed 
unreliable  because  they  are  not  consistently  used.  The  tables  pertaining  to  counseling  do  not  distinguish  educational 
counseling  from  other  types  of  counseling  and  are  not  consistently  populated  at  the  sites.  Of  a  total  of  68  tables  in 
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Although  potentially  flawed,  EDMIS  is  the  only  source  of  automated  objective  (non-self- 
reported)  data  on  participation  in  continuing  education.  At  this  time,  we  do  not  know  the  quality 
of  the  EDMIS  data  in  which  we  are  interested  for  the  evaluation.  The  only  other  potential  source 
of  objective  data  is  hardcopy  records,  which  may  not  exist,  and  which  would  present  even 
greater  coding  problems.  Although  we  wish  that  the  EDMIS  data  were  more  amenable  to 
statistical  scrutiny,  the  fact  is  that  EDMIS  is  the  only  feasible  choice  of  objective  participation 
data  for  the  evaluation. 

SOCAD  Student  Agreement  Data.  Soldier-level  data  on  SOCAD  contracts  have  been 
maintained  by  the  Servicemembers  Opportunity  Colleges  (SOC)  since  the  program  began  in  the 
1980s.  The  data  are  stored  in  two  databases  called  SOCAD-2  and  SOCAD-4  that  contain  soldier- 
level  data  on  2-  and  4-year  SOCAD  contracts,  respectively.  Data  are  sent  from  SOCAD  schools 
on  a  quarterly  basis  and  are  key-entered  manually.63  These  databases  contain  data  on  the 
SOCAD  schools  attended  and  the  date  of  the  SOCAD  agreements.64  These  data  do  not  contain 
information  on  courses  taken  or  degrees  received.  The  file  may  contain  multiple  records  per 
soldier  since  soldiers  can  enter  into  2-  and  4-year  SOCAD  agreements  and  may  also  update 
existing  agreements.65 

Administrative  Data 

Administrative  data  describing  soldier  demographic  attributes  (e.g.,  gender,  race,  marital 
status)  and  military  occupational  characteristics  (e.g.,  enlistment  term,  MOS,  separation  program 
designator)  are  recorded  in  a  number  of  databases.  The  origin  of  all  these  data,  in  some  part,  can 
be  traced  to  the  Standard  Installation/Division  Personnel  System  3  (SIDPERS3)  or  a  predecessor 
to  SIDPERS3. 

SIDPERS3.  SIDPERS3  is  the  Army’s  operational  personnel  database  installed  and 
maintained  at  the  installation-level.  SIDPERS3  provides  a  snapshot  of  the  characteristics  of 
servicemembers  at  each  Army  station  at  a  particular  point  in  time.  Although  some  variables  are 
stored  in  a  way  that  supports  longitudinal  analysis,66  SIDPERS3  data  are  not  longitudinal.  In  its 


the  EDMIS  database,  24  (35%)  were  of  initial  interest  to  this  evaluation.  Of  these,  12  (50%)  were  deemed  unreliable 
by  PERSCOM  and  an  additional  4  (17%)  were  determined  either  to  contain  information  duplicated  in  other  tables  or 
to  be  difficult  to  query.  In  general,  the  quality  of  the  remaining  selected  EDMIS  data  is  currently  unknown. 

63  The  error  rate  has  been  determined  for  some  fields  to  be  between  1%  and  2%.  The  error  rate  for  SSN  Is  unknown. 
In  addition,  rank  at  the  time  of  the  SOCAD  agreement  is  missing  approximately  33%  of  the  time. 

64  Unfortunately,  the  SOCAD  date  field  may  contain  either  the  date  of  the  SOCAD  agreement  or  the  date  that  the 
information  was  entered  on  to  the  SOCAD-2  and  SOCAD-4  databases. 

65  The  evaluation  study  will  collect  all  available  SOCAD  data  regardless  of  date,  since  once  a  SOCAD  agreement  is 
in  place,  it  exists  until  it  is  amended  or  a  degree  is  earned. 

66  For  example,  SIDPERS3  allows  for  recording  more  than  one  Armed  Forces  Classification  Test  (AFCT)  score. 

The  AFCT  is  an  in-service  test  that  evaluates  individuals  on  the  same  10  aptitude  areas  as  the  enlistment  Armed 
Services  Vocational  Aptitude  Battery  (ASVAB).  Although  there  is  a  “place”  for  more  than  1  AFCT  score,  these 
scores  are  not  routinely  input  into  SIDPERS3. 
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normal  operational  use,  the  values  of  many  SIDPERS3  variables  are  systematically  overwritten 
with  updated  information.  In  addition,  SIDPERS3  data  are  not  systematically  archived,  so  that 
there  is  no  historical  view  of  SIDPERS3  “snapshots.”  Also,  because  SIDPERS3  is  an 
operational,  installation-level  database,  there  appears  to  be  a  greater  amount  of  data 
inconsistency  between  facilities  than  one  might  find  in  a  more  centralized  system.67 

Total  Army  Personnel  Database  (TAPDB).  Installation-level  SIDPERS3  data  are 
transmitted  daily  to  PERSCOM,  where  they  are  “cleaned”  and,  in  some  cases,  modified  (e.g., 
recoded)  for  inclusion  in  the  TAPDB,  the  centralized  personnel  database  for  the  Army.  68  The 
TAPDB  contains  standardized  data  across  Army  components.  It  is  composed  of  Active  Enlisted, 
Active  Officer,  Reserve,  and  National  Guard  data.  TAPDB  is  currently  a  CA  Datacom 
mainframe  database69  that  is  being  converted  to  an  Oracle  relational  database  on  a  client  server 
platform.  Like  SIDPERS3,  the  Active  Enlisted  data  file  in  the  TAPDB,  called  the  TABDB-AE, 
contains  information  on  enlisted  soldiers  who  are  current  Army  servicemembers.  Soldiers  who 
separate  from  the  Army  are  moved  out  of  the  TABDB-AE  to  the  TAPDB  Reserve  file  120  days 
after  separation70.  Because  data  are  systematically  overwritten  with  updated  information71,  the 
TAPDB  itself  is  not  longitudinal.  Nor  is  there  a  formal,  complete  archive  of  the  TAPDB  itself, 
although  a  subset  of  TABDB-AE  data  that  is  retrofitted  to  look  like  the  Enlisted  Master  File 
(EMF),  the  Army’s  enlisted  personnel  database  prior  to  TAPDB,  is  archived  at  PERSCOM.72  In 
addition,  TAPDB  data  extracts  are  transmitted  to  other  government  organizations  -  such  as  the 
Defense  Manpower  Data  Center  (DMDC)  and  the  Army  Research  Institute  (ARI)  -  who  then 
routinely  archive  those  extracts. 

DMDC  Active  Duty  Master  and  Loss  Files.  Directed  by  DODI  1336-5,  PERSCOM 
transmits  selected  TAPDB  data  elements  to  the  DMDC  on  a  monthly  basis.73  In  the  past,  in  order 
to  conform  to  DOD-wide  standards,  DMDC  would  recode  certain  variables  and  add  others  to  the 
data  received  from  PERSCOM.  Since  September  1999,  however,  the  data  received  from  the 
services  must  conform  to  DOD-wide  standards  prior  to  transmittal  to  DMDC.  DMDC  archives 


67  This  observation  is  based  on  conversations  with  PERSCOM  TAPDB  specialists  on  23  March  2000. 

68  While  SIDPERS3  is  the  primary  input  to  TAPDB,  there  are  a  number  of  others.  For  example,  the  Army 
Recruiting  Accession  Data  System  (ARADS)  provides  the  initial  recruiting  information  on  soldiers  to  the  TAPDB. 

69  CA  Datacom  is  relational  database  software  developed  by  Computer  Associates  for  use  on  mainframe  computers. 

70  The  Reserve  File,  maintained  in  St.  Louis,  records  soldier  attributes  at  the  time  of  separation. 

71  Like  SIDPERS3,  some  TAPDB  variables  are  stored  in  a  way  that  makes  longitudinal  analysis  possible.  For 
example,  TAPDB  maintains  data  on  a  soldier’s  last  20  job  assignments.  However,  other  variables,  such  as  education 
level,  are  overwritten. 

72  There  are  10  years  of  monthly  EMF  data  archived  at  PERSCOM.  It  is  unclear  whether  a  current  codebook  exists 
for  the  EMF. 

73  PERSCOM  TAPDB  data  that  are  transmitted  to  DMDC  are  cut  mid-month.  PERSCOM  then  spends  about  two 
weeks  cleaning  the  data.  These  data  are  then  transmitted  to  DMDC  by  file  transfer  protocol  (FTP)  by  around  the  first 
of  the  following  month.  Servicemember  loss  information  is  transmitted  to  DMDC  on  a  weekly  basis. 
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each  month’s  74  “snapshot”  from  the  TAPDB,  effectively  creating  a  longitudinal  database  in  the 
process.  These  data  are  called  the  Active  Duty  Master  and  Loss  Files  (ADMLF).  Stringing 
closely  timed  TAPDB  “snapshots”  together  creates  a  longitudinal  database  where  changes 
occurring  in  soldier  attributes  over  time  are  observable. 

Military  Entrance  Processing  Command  (MEPCOM)  data  are  matched  to  TAPDB  data 
by  DMDC.  Soldiers  appear  in  the  DMDC  Active  Duty  Master  File  in  the  archive  months  during 
which  they  are  on  active  duty.  Once  a  soldier  separates,  his  or  her  record  is  dropped  from  that 
month’s  Master  data  and  appears  in  the  DMDC  Loss  File. 

DMDC  Cohort  File.  There  are  a  number  of  DMDC  files  that  are  based  in  whole  or  in  part 
on  the  ADMLF.  One  such  file  that  has  been  used  repeatedly  in  previous  ARI  research  is  the 
Active  Duty  Military  Enlisted  Cohort  File.  The  Cohort  file  is  composed  of  an  accession  year 
cohort  as  identified  by  the  MEPCOM  Examination  and  Accession  File.  It  is  then  matched  on  a 
yearly  basis  with  fiscal  year-end  ADMLF  data  to  track  individuals’  military  careers.  The  Cohort 
file  basically  consists  of  four  blocks  or  types  of  data.  The  first  block  contains  MEPCOM 
accession  data,  such  as  the  AFQT  score  and  entry  date.  The  next  block  consists  of  information 
extracted  from  a  match  to  the  Master  or  Loss  File,  whichever  is  the  most  recent.  Therefore,  if  a 
soldier  is  active,  the  “most  recent”  block  will  contain  selected  variables  from  the  most  recent 
Master  File;  if  a  soldier  has  separated,  data  in  the  “most  recent”  block  in  the  Cohort  File  will 
come  from  the  Loss  File.  The  third  Cohort  block  of  information  records  data  from  the  second 
most  recent  match  to  either  the  Master  or  Loss  File.  Finally,  the  fourth  block  contains  data  from 
the  first  Loss  File  match,  where  applicable.  Although  it  provides  active  duty  information  at  two 
or  three  points  in  time  (most  recent,  second  most  recent,  and  first  loss),  the  Cohort  file  is  not 
longitudinal.  Since  the  Cohort  file  only  includes  recent  (most  and  second  most  recent) 
Master/Loss  data,  more  longitudinal  information  is  lost  as  soldier  tenure  increases. 

EMF.  A  quarterly  EMF,  which  is  transmitted  to  ARI,  contains  a  subset  of  variables  found 
in  the  larger  PERSCOM-maintained  EMF  discussed  above.  The  “ARI  EMF”  is  transmitted  as  a 
flat  file,  which  is  read,  processed,  and  archived  by  ARI.  Taken  in  aggregate,  the  quarterly 
archives  of  the  “ARI  EMF’  data  provide  longitudinal  information  on  soldier  attributes  and 
military  job  characteristics. 

ARI  Special  Attrition  Databases.  The  First  Tour  Attrition  Database,  based  on  the  FY 
1992  DMDC  Cohort  File,  was  constructed  for  ARI  in  1998  as  a  prototype  database  for  the  study 
of  attrition.  A  subsequent  ARI  attrition  project  has  resulted  in  the  development  of  an  extensive 
database  on  the  FY  1999  accession  cohort  of  enlisted  soldiers  (“Project  First-Term”).  The  data  in 
this  1999  attrition  study  are  current  as  of  December  1999  and  there  are  plans  to  update  these  data 
through  the  end  of  FY  2003.  The  Project  First-Term  data  are  especially  rich  in  administrative  and 
attitudinal  data.  Both  attrition  databases  contain  quarterly  EMF  data,  providing  longitudinal 
information  on  cohort  members. 

DMDC  Special  Cohort  and  Continuer  Files  (DSCAC).  The  DSCAC  data  are  based  on 
fiscal  year  accession  cohorts  and  contain  data  that  track  the  careers  of  active  duty  enlisted 


74  Monthly  data  are  archived  from  December  1992  to  the  present;  quarterly  data  are  available  since  June  1975. 
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servicemembers,  including  soldiers  with  prior  service,  who  enter  the  military  through  the 
MEPCOM.  Using  MEPCOM  and  ADMLF  data,  an  individual  is  tracked  quarterly  for  the  first 
four  and  a  half  years  and  then  every  six  months  until  the  soldier  has  been  followed  for  20  years 
(or  through  the  most  recent  file).  Hence,  the  DSCAC  data  are  longitudinal.  Each  record  contains 
four  main  blocks  of  information:  (a)  MEPCOM  accession  data,  (b)  first  loss  information, 

(c)  active  duty  data  representing  a  snapshot  of  the  soldiers’  career  status  at  a  given  point  in  time, 
and  (d)  DMDC-constructed  “flag”  information  which  makes  the  files  easier  to  use.  The  DSCAC 
database  currently  contains  data  on  20  separate  cohorts,  beginning  in  1978  through  1997. 

The  Active  Duty  Personnel  Edit  File  ( PEF ),  The  PEF  is  also  based  on  the  ADMLF,  but  is 
created  by  DMDC  on  an  ad  hoc  basis.  Based  on  a  data  request  like  that  for  this  evaluation  study, 
DMDC  can  extract  the  population  and  data  elements  of  interest  and  can  provide  “snap-shots”  of 
soldier  attributes  on  a  monthly,  quarterly,  or  year-end  basis,  or  whenever  a  change  in  a  specified 
variable  (e.g.,  pay  grade)  occurs.  PEF  data  are,  therefore,  longitudinal. 

Defense  Enrollment  Eligibility  Reporting  System  (DEERS).  The  DEERS  database 
contains  one  record  for  each  servicemember  whether  on  active  duty  for  more  than  30  days, 
retired,  or  a  member  of  the  Reserve  component.  DEERS  also  contains  a  record  for  each  of  the 
servicemember’ s  family  members.  Servicemember  and  family  data  include:  (a)  personal  data, 
such  as  SSN,  date  of  birth  and  gender;  (b)  service-related  information,  such  as  service  and  unit 
identification  code;  (e)  geographic  data,  and  (d)  information  on  benefits,  such  as  Montgomery  GI 
Bill  (MGIB)  education  benefits. 

Self-Reported  Education  and  Attitudinal  Data 

There  are  two  sources  of  self-reported  education  and  attitudinal  data:  the  Survey  of 
Active  Duty  Personnel  (SADP)  and  the  Sample  Survey  of  Military  Personnel  (SSMP). 

SADP.  This  survey  is  administered  to  a  stratified  random  sample  of  DoD  members 
approximately  every  four  years.  The  1999  survey  was  administered  between  September  1999 
and  January  2000  and  asks  soldiers  to  report  whether  in  the  past  12  months  they  participated  in 
(a)  adult  continuing  education  or  counseling,  (b)  tuition  assistance  programs  for  college, 

(c)  technical  or  vocational  programs,  and/or  (d)  a  basic  skills  education  program.  The  survey 
contains  information  regarding  the  intentions  of  soldiers  to  remain  in  the  military,  their 
satisfaction  with  off  duty  educational  opportunities,  primary  reasons  why  the  member  joined  the 
military  (e.g.,  for  education  benefits  and  opportunities),  and  other  demographic  and  job  related 
characteristics  that  are  of  interest  to  this  evaluation.  While  the  1999  SADP  collected  SSN  from 
respondents,  the  survey  data  will  not  be  released  containing  SSN  for  reasons  of  confidentiality 
and  data  security.  There  is  currently  no  codebook  available  describing  1999  SADP  data 
elements. 

SSMP.  The  SSMP  containing  ACES  participation  data  is  administered  by  ARI  to  a 
random  sample  of  Army  enlisted  soldiers  and  officers  every  two  years.  The  survey  contains  such 
items  as  satisfaction  with  military  life  and  education  level  and  intention  to  reenlist,  as  well  as 
questions  about  participation  in  ACES  programs.  Unlike  the  SADP,  the  SSMP  is  anonymous,  in 
that  no  unique  soldier  identifier,  such  as  SSN,  is  collected  by  the  survey. 
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Economic  Data 


The  Bureau  of  Labor  Statistics  (BLS)  publishes  information  on  unemployment  rates  and 
pay  and  benefits.  This  information  is  readily  available  on  the  BLS  website. 

Evaluation  of  Data  Sources 

What  are  the  optimal  sources  of  data  for  the  study  described  in  the  evaluation  plan?  A 
discussion  of  the  “goodness  of  fit”  between  the  evaluation  study  requirements  and  the 
characteristics  of  the  data  sources  considered  is  provided  below. 

ACES  Participation  Data.  The  EDMIS  and  SOCAD  contract  data  are  the  only  electronic 
data  sources  on  objective  education  participation  in  the  programs  of  interest  to  this  evaluation. 
As  such,  they  are  key  data  sources  in  the  evaluation  study  data  development  plan.  Selecting 
education  participation  from  January  1999  forward  for  the  main  analyses  and  eliminating 
soldiers  who  were  in  sites  that  did  not  have  EDMIS  installed  by  that  date75  satisfies  the  criterion 
of  data  completeness.  The  further  requirement  of  the  availability  of  longitudinal  time-varying 
data  is  also  satisfied  by  EDMIS  and  SOCAD  because  historical  data  are  currently  available  on¬ 
line  or  will  be  available  in  archive  in  the  future.  In  addition,  these  data  contain  SSN,  satisfying 
the  requirement  for  a  common  link  variable. 

Administrative  Data.  Three  sources  of  administrative,  demographic,  and  military  career 
data  can  be  eliminated  from  further  consideration  because  they  do  not  contain  longitudinal  data 
for  time-varying  attributes:  SIDPERS3,  TAPDB,  and  the  DMDC  Cohort  file.  The  DMDC 
ADMLF  can  also  be  eliminated  because  it  does  not  satisfy  the  criterion  of  efficiency.  The  sheer 
number  of  monthly  files  and  the  fact  that  the  Master  and  Loss  components  of  the  ADMLF  are 
physically  distinct  files  that  must  be  combined  to  obtain  full  career  histories,  will  make  working 
with  the  ADMLF  cumbersome  and  inefficient,  particularly  in  the  face  of  more  attractive 
alternatives. 

Neither  the  DSCAC  nor  the  Cohort  data  meet  the  criterion  of  efficiency  for  the 
reenlistment  and  the  promotion  analyses.  This  is  because  the  DSCAC  and  Cohort  files  are  based 
on  accession  year  cohorts,  whereas  the  retention  and  promotion  analyses  will  be  conducted  on 
event-year  cohorts  -  the  year  in  which  an  event  occurred  (i.e.,  promotion)  or  is  scheduled  to 
occur  (i.e.,  ETS).  Substantial  time  and  effort  are  required  to  identify  an  event-based  cohort  from 
accession-based  cohort  data. 

The  1992  First  Tour  Attrition  Database  does  not  provide  data  on  the  populations 
specified  in  the  evaluation  plan  and  is,  therefore,  eliminated  from  further  consideration.  Because 
complete  EDMIS  data  are  available  starting  in  January  1999,  educational  participation  data  are 
available  about  seven  years  into  the  careers  of  the  1992  First  Tour  Attrition  Database  cohort.  By 
that  time,  most  soldiers  are  either  separated  or  well  into  their  second  tour  of  service.  Therefore, 


To  ensure  complete  EDMIS  data  for  all  cases,  soldiers  who  were  ever  assigned  to  any  of  the  seven  sites  that 
currently  do  not  have  EDMIS  installed  or  Ft.  Buchanan,  where  EDMIS  was  installed  in  September  1999,  will  be 
eliminated  entirely  from  the  evaluation  study. 
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the  1992  First  Tour  Attrition  cohort  is  too  old  for  purposes  of  this  evaluation.  On  the  other  hand, 
while  the  Project  First-Term  cohort  is  somewhat  too  young  to  measure  first  term  attrition,  it  is  an 
extremely  rich  source  of  data  for  observing  early  attrition  and  reclassification.  The  only  required 
variable  that  is  missing  from  the  Project  First-Term  data  is  MGIB. 

The  EMF  (both  the  ARI  and  the  larger  PERSCOM  EMF)  and  the  PEF  contain 
longitudinal  data  on  the  populations  of  interest  to  this  evaluation.  Because  the  EMF  and  PEF  data 
are  both  based  on  snap-shots  of  the  force  at  given  points  in  time,  the  level  of  effort  required  to 
identify  cohort  members  and  extract  their  data  are  very  similar.76  While  the  PEF  contains  all  the 
variables  required  by  the  evaluation  plan,  the  “ARI  EMF”  lacks  a  critical  variable  -  the  unit 
identification  code  (UIC).  The  “ARI  EMF’  can,  therefore,  be  dropped  from  further 
consideration. 

The  remaining  two  administrative  data  sources  are  attractive  candidates  for  identifying 
members  of  the  reenlistment  and  promotion  cohorts.  There  is  currently  no  reason  to  believe  that 
costs  associated  with  preparing  the  data  for  analysis  (e.g.,  data  cleaning,  recoding,  diagnostics) 
will  differ  substantially  for  data  extracted  from  the  larger  EMF  and  those  culled  from  the  PEF. 
The  EMF  contains  data  on  the  MGIB,  a  static  variable,  which  the  PEF  lacks.  However,  this 
information  could  be  obtained  from  DEERS  for  the  attrition/reclassification  and  reenlistment 
cohorts.77  Both  the  EMF  and  the  PEF/DEERS  could  be  used  to  supply  the  administrative  data  for 
the  reenlistment  and  promotion  analyses.  However,  whereas  DMDC  produces  ad  hoc  files 
routinely  and  is,  therefore,  experienced  in  filling  non-standard  data  requests,  PERSCOM  does 
not  often  fill  non-routine  data  requests  from  the  EMF.78  Therefore,  for  reason  of  efficiency,  we 
recommend  using  PEF/DEERS  data  as  the  source  for  administrative  data  on  demographic 
attributes  and  military  characteristics  for  the  reenlistment  and  promotion  analyses.  In  addition, 
DEERS  can  be  used  to  supply  MGIB  to  the  Project  First-Term  data. 

What  frequency  of  data  “snap-shots”  is  sufficient  to  provide  a  longitudinal  perspective  of 
servicemember  careers  and  behavior?  The  types  of  time-varying  variables  of  interest  to  the 
evaluation  study  -  such  as  marital  status,  number  of  dependents,  MOS,  pay  grade  -  typically  do 
not  change  routinely  or  often.  Monthly  snap-shots  are  probably  too  frequent,  while  annual  files 
are  not  frequent  enough.  Therefore,  data  consisting  of  quarterly  snap-shots  over  the  observation 
period  are  recommended.  In  terms  of  the  early  attrition  and  reclassification  analyses,  the  plans 
for  Project  First-Term  data  already  include  quarterly  EMF  updates  through  FY  2003.  We 
recommend  quarterly  PEF  data  for  the  reenlistment  and  promotion  cohorts. 

Self-Reported  Education  and  Attitudinal  Data.  Although  the  SSMP  collects  self-reported 
education  participation  data  and  attitudinal  measures,  its  lack  of  respondent  SSNs  precludes  its 


76  As  discussed  above,  the  level  of  effort  required  to  identify  event  year  cohort  members  is  substantially  less  from 
“snap-shot”  files  like  the  EMF  and  the  PEF  than  for  accession  year  based  files  like  the  Cohort  file. 

77  MGIB  data  is  not  relevant  for  the  promotion  analyses. 

?SThis  observation  is  based  on  conversations  with  Alex  Schneider  and  Don  Edwards,  Requirements  Team  Plans 
Branch,  PERSINSD,  PERSCOM. 
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linkage  to  any  other  data.  The  SADP  is  the  only  known  source  for  attitudinal  measures  and  self- 
reported  education  that  requests  SSN  from  respondents.  In  addition,  it  was  administered  at 
approximately  the  period  of  time  that  is  appropriate  to  the  evaluation  study.79  Since  SADP  SSNs 
will  be  withheld  from  evaluation  study  contractors  for  reasons  of  confidentiality  and  privacy,  a 
strategy  of  encrypting  SSNs,  described  in  a  later  discussion,  has  been  developed  to  circumvent 
this  issue. 

In  summary,  the  following  soldier-level  data  sources  have  been  selected  for  the  ACES 
evaluation  study: 

•  EDMIS  for  ACES  participation  data, 

•  SOCAD-2  and  SOCAD-4  for  data  on  SOCAD  Student  Agreements, 

•  PEF/DEERS  for  administrative  demographic  and  military  data  for  the  reenlistment  and 
promotion/reclassification  cohorts, 

•  Project  First-Term/DEERS  for  administrative  demographic  and  military  data  for  the  early 
attrition/reclassification  cohorts,  and 

•  SADP  for  attitudinal  and  self-reported  education  participation  data. 

Table  4  presents  a  summary  of  the  limitations  of  the  rejected  data  sources  by  the  data 
requirement  criteria  discussed  above.  Because  all  sources  of  automated  education  participation 
data  of  interest  have  been  selected  (i.e.,  EDMIS  and  SOCAD-2  and  -4),  the  requirement  for 
complete  education  data  is  not  included  in  Table  16. 


Table  16 

Table  of  Limitations  of  Rejected  Data  Sources  by  Data  Requirement  Criterion 


■  0~s 

Data  Requirement  Criterion  "  ~T! 

vmmS- 

Population(s) 

Efficient  of  Genre 

Un^aL 

SIDPERS3 

X 

TAPDB 

X 

ADMLF 

X 

DMDC  Cohort 

X 

“ARIEMF* 

X 

“PERSCOM  EMF’ 

X 

1st  Tour  Attrition  Database 

X 

DSCAC 

X 

SSMP 

X 

79  Because  the  exact  date  that  individual  respondents  took  the  SADP  is  unknown,  only  criterion  measures  occurring 
after  January  2000  (the  latest  administration  date  for  the  1999  SADP)  can  be  analyzed  using  explanatory  measures 
from  the  1999  SADP. 
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Producing  an  Evaluation  Analytic  Database 
Overview  of  the  Data  Collection  Process 

Data  Encryption.  One  of  the  first  steps  in  building  an  evaluation  analytic  database  is  to 
identify  cohort  members.  DMDC  will  identify  the  reenlistment  and  promotion  cohorts  from  the 
PEF  administrative  data  and  extract  their  PEF/DEERS  data.  Once  the  data  are  obtained,  the 
contractor  will  assess  whether  the  cohorts  appear  to  be  correctly  identified.80  Next,  MGIB  from 
DEERS  data  must  be  produced  for  the  Project  First-Term  cohort. 

Because  of  confidentiality  issues  with  the  1999  SADP,  special  care  will  be  taken  to  strip 
SSNs  from  data  provided  to  the  evaluation  contractor  that  will  be  linked  to  the  SADP.  DMDC 
will  encrypt  the  SSN  of  each  member  of  the  reenlistment  and  promotion  cohorts  and  will  provide 
the  contractor  with  the  PEF/DEERS  data  identified  with  the  encrypted,  rather  than  unencrypted, 
SSNs.  In  addition  to  this  PEF/DEERS  data,  DMDC  will  create  electronic  ASCII  files  containing 
only  two  data  elements:  the  encrypted  and  unencrypted  SSNs  of  the  reenlistment  and  promotion 
cohort  members.  DMDC  will  transmit  a  copy  of  these  electronic  files  to  the  evaluation  project 
COTR.  The  COTR  will,  in  turn,  send  a  copy  of  these  files  containing  the  SSNs  to  the  appropriate 
POCs  for  matching  to  EDMIS  and  SOCAD  data  on  the  basis  of  unencrypted  SSN.  The  parties 
responsible  for  matching  cohort  members’  unencrypted  SSNs  to  the  EDMIS  and  SOCAD  data 
will  be  asked  by  the  project  COTR  to  drop  the  unencrypted  SSN  from  the  data  returned  to  the 
contractor  and  include  only  encrypted  SSN  as  the  unique  identifier.  DMDC  will  also  apply  the 
same  encryption  formula  to  the  1999  SADP  and  provide  the  contractor  with  the  survey  data  and 
encrypted  SSN.  Since  the  Project  First-Term  cohort  will  not  be  linked  to  the  SADP,  EDMIS  data 
will  be  obtained  for  the  cohort  members  by  SSN. 

Once  data  cleaning  and  manipulation  are  completed  (see  below),  the  EDMIS  and 
SOCAD  data  can  be  merged  to  PEF/DEERS  by  encrypted  SSN  and  to  Project  First- 
Term/DEERS  by  unencrypted  SSN  to  form  preliminary  analytic  databases.  The  SADP  data  form 
the  remaining  main  cohort.  The  SADP  cohort  will  be  merged  with  the  reenlistment  and 
promotion  cohorts  on  the  basis  of  encrypted  SSN  to  form  the  intersected  cohorts  described 
above.81 


Although  the  project  COTR  will  provide  the  encrypted  and  unencrypted  SSNs  to  the 
appropriate  parties  for  obtaining  EDMIS  and  SOCAD  data,  the  contractor  must  provide  detailed 
specifications  of  the  data  required.  Specifications  for  obtaining  the  evaluation  study  data  are 
presented  below.  Access  to  Project  First-Term  data  is  rather  straightforward  and  involves  a 
simple  authorization  from  ARI  for  use  of  the  data.  For  this  reason,  data  request  specifications  for 
Project  First-Term  data  are  not  provided  here. 


80  This  assessment  can  be  made  by  generating  frequency  distributions  or  crosstabulations  on  those  variables  that 
define  the  cohort. 

This  encryption  strategy  is  fairly  complex  and  somewhat  onerous.  However,  if  SADP  confidentiality  requirements 
remain  unchanged  and  the  SADP  must  be  linked  to  other  data  sources,  this  approach  is  reasonable.  If  confidentiality 
requirements  are  relaxed  or  the  SADP  is  analyzed  as  stand-alone  data,  this  encryption  strategy  can  be  greatly 
simplified  or  eliminated  entirely. 
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PEF/DEERS  Data  Specifications.  DMDC  requires  a  written  memorandum  by  the 
organization  requesting  data  stating  the  specific  data  file(s)  of  interest,  the  use  that  the  data  will 
be  put  to,  confidentiality  and  security  precautions  that  will  be  undertaken,  the  population  and 
variables  needed,  and  any  special  instructions  or  requirements.  DMDC  requires  approximately  6 
weeks  to  produce  ad  hoc  PEF  files  based  on  non-accession  year  cohorts, 

EDMIS  Data  Specifications.  The  DMDC  PEF/DEERS  and  the  Project  First- 
Term/DEERS  data  will  define  the  observations  for  which  EDMIS  data  will  be  extracted.  As 
stated  above,  the  encrypted  and  unencrypted  SSNs  from  the  PEF/DEERS  will  be  provided  to  the 
designated  EDMIS  POC82  by  the  project  COTR  for  the  extraction  of  EDMIS  data.  The  SSNs 
from  the  Project  First-Term  cohort  will  also  be  provided  to  the  designated  EDMIS  POC. 

SOCAD-2  and  SOCAD-4  Data  Specifications.  Like  EDMIS,  the  encrypted  and 
unencrypted  SSNs  from  the  PEF/DEERS  and  the  SSNs  from  the  Project  First-Term  cohort  will 
be  provided  to  the  designated  SOCAD  POC83  by  the  project  COTR. 

SADP  Data  Specifications.  As  noted  above,  the  SADP  defines  its  own  cohort  in  the 
evaluation  study. 

Procedures  for  Creating  an  Evaluation  Analysis  File 

Data  Diagnostics  and  Cleaning.  Statistical  Analysis  System  (SAS)  or  the  Statistical 
Package  for  the  Social  Sciences  (SPSS)  system  files  should  be  created  for  all  evaluation  study 
data.  First,  the  contractor  conducting  the  evaluation  will  review  all  documentation  accompanying 
the  data.  Second,  in  order  to  become  familiar  with  the  data,  the  contractor  will  conduct  a 
preliminary  review  of  univariate  statistics  (frequency  distributions  for  discrete  variables  and 
means/standard  deviations  for  continuous  data  elements)  of  all  data.  Third,  based  in  part  on  the 
review  of  univariate  statistics,  the  contractor  will  clean  and  edit  the  data.  Careful  attention  should 
be  paid  to  the  coding  of  missing  values.  For  example,  some  DMDC  data  elements  are  coded  0  to 
indicate  missing  data.  Since  the  evaluation  data  will  be  subjected  to  statistical  scrutiny,  cases 
with  numerically  coded  missing  data  must  be  recoded  to  system  missing  values.  Variables  whose 
values  fall  outside  the  acceptable  range  will  be  identified  and  “cleaned”.  The  logical  consistency 
between  variables  will  be  assessed.  This  can  be  done  through  a  series  of  cross-tabulations  and 


82  There  are  a  number  of  unknowns  at  the  time  of  this  writing  regarding  the  procurement  of  EDMIS  data.  First,  the 
specific  individual  or  individuals  at  PERSCOM  who  will  be  responsible  for  obtaining  EDMIS  data  have  not  been 
identified.  Second,  although  it  is  clear  that  the  contractor  conducting  the  evaluation  study  will  not  have  hard-wired 
or  network  access  to  EDMIS  data,  the  process  by  which  the  information  will  be  obtained  is  otherwise  also  unknown. 
The  most  recent  information  indicates  that  EDMIS  data  requests  made  by  the  contractor  may  be  forwarded  by 
PERSCOM  to  individual  education  centers  that  will  be  responsible  for  the  processing  of  that  information.  Third,  the 
form  that  the  request  for  data  should  take  is  also  unclear.  The  assumption  here  is  that  a  written  request  for  data  is 
sufficient.  Since  they  will  not  have  access  to  the  data,  the  contractor’ s  role  in  this  process  is  limited  to  requesting 
data  rather  than  procuring  it.  Hence,  the  relative  merits  of  one  strategy  of  data  procurement  over  another  could  not 
be  ascertained. 

83  Information  obtained  from  SOC  indicates  that  the  request  for  data  should  be  sent  to  PERSCOM,  which  will 
forward  the  request  to  SOC  for  fulfillment.  The  specific  individual  at  PERSCOM  who  will  receive  and  forward  the 
request  has  not  been  identified  as  of  the  time  of  this  writing. 
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programming  statements  designed  to  isolate  the  relationship  of  two  or  more  variables.  Skip 
patterns  in  the  S  ADP  will  be  identified  and  checked  to  ensure  that  the  electronic  data  reflect  the 
established  patterns  of  response  in  the  survey.  All  edits  will  be  checked  to  ensure  that  the  desired 
changes  have  been  made.  Finally,  all  edits  to  data  will  be  made  programmatically,  rather  than 
interactively,  and  all  programs  must  be  retained  for  purposes  of  documentation.  All  variables 
will  be  assigned  value  labels  or  formats  to  facilitate  ease  of  use. 

Data  Structure.  Once  the  data  have  been  cleaned,  the  issue  of  the  structure  of  the 
evaluation  data  can  be  addressed.  Data  from  the  PEF/DEERS,  Project  First-Term/DEERS,  and 
the  S  ADP  contain  one  record  per  soldier.  EDMIS  data  may  contain  multiple  records  for  an 
individual  soldier  within  a  facility  and  the  soldier  may  appear  in  the  EDMIS  records  from  many 
facilities.  The  combined  SOCAD-2  and  -4  data  may  also  contain  more  than  one  record  per 
soldier.  The  goal  for  statistical  processing  is  to  arrive  at  a  file  for  each  type  of  analysis  (early 
attrition,  reenlistment,  reclassification,  and  promotion)  that  contains  a  single  record  per  soldier. 

EDMIS  data  can  be  combined  with  SOCAD  data  to  form  a  data  file  containing  education 
participation  for  all  ACES  programs  of  interest.84  Because  the  criterion  measures  must  always 
post-date  the  explanatory  variables,  education  participation  must  always  occur  before  the  early 
attrition,  reenlistment,  reclassification,  or  promotion  being  analyzed.  One  way  to  ensure  that  the 
education  participation  variables  are  temporally  correct  for  a  particular  criterion  is  to  merge  the 
PEF  data  containing  the  date  of  the  criterion  of  interest  with  the  combined  EDMIS/SOCAD  data 
containing  multiple  records  per  SSN  across  all  facilities.  The  date  of  the  criterion  measure  can 
then  be  compared  to  the  date  of  the  education  participation,  and  only  the  temporally  correct 
education  records  can  be  retained.  Those  “correct”  EDMIS  records  can  then  be  combined  to 
form  a  single  record  containing  blocks  of  data  arranged  chronologically  describing  education 
participation  during  the  observation  period  of  the  criterion.  This  single  EDMIS  record  must  then 
be  merged  to  the  full  set  of  administrative  PEF/DEERS  and  Project  First-Term/DEERS  data.  It 
must  be  remembered  that  the  temporally  correct  education  participation  must  be  identified  for 
each  criterion  measure  analyzed.” 

Variable  Coding.  Because  codebooks  are  not  available  for  the  SADP,  EDMIS,  and 
SOCAD-2  and  4  databases,  a  discussion  of  the  coding  of  analysis  variables  is  not  possible  for 
these  data. 


If  necessary,  EDMIS  data  from  all  facilities  will  be  concatenated  to  form  a  total  EDMIS  database.  The  EDMIS 
and  SOCAD  data  can  then  be  combined  so  that  the  resultant  file  contains  both  EDMIS  and  SOCAD  variables;  the 
records  from  EDMIS  will  contain  missing  data  for  the  SOCAD  data  elements  and  vice  versa.  Variables  can  then  be 
created  that  store  analytically  relevant  data  from  the  EDMIS  and  SOC  portions  of  the  record.  An  example  of 
analytically  relevant  variable  construction  is  the  creation  of  a  variable  “DATE 44  that  assumes  the  value  of  a  course 
date  for  EDMIS  records  and  the  SOCAD  agreement  date  for  SOC  records.  Essentially  this  combined  file  simulates 
the  EDMIS  data  if  it  had  contained  reliable  SOCAD  data.  This  combined  file  can  then  be  sorted  by  SSN  and  date. 
The  resultant  file  will  contain  education  participation  data  from  all  facilities  with  multiple  records  per  soldiers  in 
chronological  order. 

There  are  other  ways  to  achieve  the  same  goal.  For  example,  a  single  soldier  EDMIS  record  can  be  created  at  each 
site  and  then  merged  together  to  form  one  comprehensive  record.  Once  merged  with  PEF  data,  the  temporally 
correct  blocks  of  data  on  this  comprehensive  record  can  then  be  identified  with  some  elegant  array  processing. 
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Summary 


The  purpose  of  this  data  development  plan  is  to  ensure  a  sound  and  successful  evaluation 
by  meeting  all  its  data  needs.  The  data  development  plan  first  identifies  the  data  and  population 
requirements  set  forth  in  the  evaluation  plan.  Next,  all  relevant  data  sources  are  reviewed  in  light 
of  these  requirements.  Third,  an  assessment  of  the  “goodness  of  fit”  between  data  characteristics 
and  these  requirements  is  presented.  This  assessment  results  in  the  selection  of  databases  for  the 
evaluation  study.  Finally,  a  discussion  is  presented  on  procedures  for  building  the  evaluation 
database,  including  data  encryption,  data  requests/specifications,  diagnostics,  data  editing, 
database  structure,  and  variable  coding. 
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SUMMARY  AND  CONCLUSIONS 


Although  past  research  investigated  a  limited  number  of  CE  programs,  the  results  of  this 
research  identified  potentially  substantial  beneficial  effects  of  military  voluntary  education 
programs  on  retention  and  performance.  These  results  also  affirmed  the  importance  of 
considering  selection  bias  in  any  evaluation  of  CE  programs  because  the  individuals  who 
participate  in  these  programs  tend  to  be  better  qualified  and  more  highly  motivated  than  those 
who  don’t  participate.  In  addition,  previous  studies  identified  many  of  the  other  explanatory 
factors  that  should  be  included  as  control  variables  in  an  evaluation  of  ACES  to  avoid  misleading 
results. 


The  evaluation  and  database  development  plans  contained  in  this  report  provide  a 
procedure  to  determine  whether  the  effects  found  in  previous  research  studies  occur  for  the 
Army’s  ACES  program,  as  well.  The  plans  build  on  the  results  of  past  research  to  isolate  the 
effects  of  participation  in  ACES  programs  from  the  effects  of  other  potentially  confounding 
variables.  They  also  consider  limitations  on  the  availability,  accuracy,  and  completeness  of 
participation  and  outcome  data  that  make  a  comprehensive  evaluation  of  these  programs 
difficult. 

The  evaluation  plan  addresses  the  following  five  ACES  programs:  (a)  TA,  (b)  SOCAD, 
(c)  FAST,  (d)  MOS  Improvement  Courses,  and  (e)  NCO  Leader  Development  Courses.  The 
design  of  the  plan  is  based  on  a  conceptual  framework  that  allows  one  to  derive  hypotheses 
regarding  the  direct  and  indirect  effects  of  the  selected  programs  on  early  attrition,  reenlistment, 
promotion,  and  MOS  reclassification.  The  analysis  planned  for  each  outcome  measure  is  based 
on  a  specific  conceptual  model  that  enumerates  and  organizes  the  explanatory  factors  that 
contribute  to  the  outcome  being  assessed.  This  model,  along  with  the  results  of  previous 
research,  specifies  the  form  of  the  analysis  and  the  variables  to  be  incorporated  in  it.  By 
incorporating  variables  that  predict  ACES  participation  into  the  analysis,  the  plan  reduces  the 
effects  of  selection  bias  on  the  obtained  results. 

One  on  the  obstacles  that  needed  to  be  overcome  in  the  development  of  the  plan  was  the 
limited  availability  of  participation  data.  Addressing  this  limitation  required  the  specification  of 
several  analysis  cohorts,  each  designed  to  support  the  evaluation  of  the  effects  of  particular 
ACES  programs  on  one  or  two  outcome  measures.  Use  of  multiple  cohorts  necessarily  increases 
the  effort  required  to  obtain,  test,  structure,  and  analyze  data.  However,  it  allows  the  effects  of 
ACES  programs  on  all  relevant  outcome  measures  to  be  assessed  using  objective  participation 
data  that  is  only  available  for  the  most  recent  years. 

Implementing  the  evaluation  plan  will  require  overcoming  several  challenges.  The 
primary  difficulty  will  be  the  obtaining  and  structuring  data  from  EDMIS  regarding  participation 
in  ACES  programs.  Because  EDMIS  is  maintained  independently  at  more  than  130  Education 
Centers,  and  because  the  system  is  an  operational  database  that  is  not  designed  to  support 
statistical  analyses,  developing  the  analytical  database  for  the  evaluation  will  require 
considerable  effort  and  expense.  The  database  development  plan  contained  in  this  report 
anticipates  some  of  the  problems  that  may  arise,  but  it  is  likely  that  other  unanticipated  problems 
will  occur  when  the  database  is  actually  built. 
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Alternative  approaches  have  been  considered  to  reduce  the  effort  required  for  database 
development  and  to  circumvent  some  of  the  risks  of  error  or  inconsistencies  in  the  obtained 
participation  data.  Some  of  these  approaches  are  the  following: 

•  Conduct  analyses  at  selected  bases  that  represent  a  cross  section  of  Army  personnel,  thus 
reducing  the  potential  for  inconsistencies  between  sites. 

•  Reducing  the  number  of  variables  obtained  from  EDMIS  to  the  minimally  required 
variables.  This  reduction  would  still  allow  us  to  conduct  the  major  analyses  included  in 
the  plan  but  would  eliminate  the  possibility  of  conducting  ancillary  or  exploratory 
analyses. 

•  Increase  the  use  of  data  sources  that  include  self-report  participation  measures,  such  as 
the  S  ADP  or  the  SSMP. 

•  Conducting  a  relatively  small  preliminary  analysis  designed  solely  to  determine  the 
viability  of  using  EDMIS  data  for  the  evaluation  effort.  The  results  of  this  analysis  would 
provide  recommendations  about  the  quality  of  EDMIS  data  and  the  effort  required  to 
format  the  data  for  analysis. 

We  recommend  that  the  process  of  developing  alternative  approaches  that  may  be  more  cost 
effective  continue.  However,  these  approaches  should  be  based  on  the  general  guidance  provided 
by  the  conceptual  models  described  in  the  evaluation  plan. 
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APPENDIX  B 


LIST  OF  ACRONYMS 


Acronym 

Definition 

AARTS 

Army/American  Council  on  Education  Registry  Transcript  System 

ACES 

Army  Continuing  Education  System 

ACF 

Army  College  Fund 

ACT 

American  College  Test 

ADMLF 

Active  Duty  Master  and  Loss  Files 

AFQT 

Armed  Forces  Qualification  test 

APTP 

The  Army  Personnel  Testing  Program 

ARI 

Army  Research  Institute 

BLS 

Bureau  of  Labor  Statistics 

BSEP 

Basic  Skills  Education  Program 

CCAF 

Community  College  of  the  Air  Force 

CE 

Continuing  Education 

CLEP 

College  Level  Examination  Program 

CONUS 

Continental  United  States 

CY 

Calendar  Year 

DANTES 

Defense  Activity  for  Nontraditional  Education  Support 

DEERS 

Defense  Enrollment  Eligibility  Reporting  System 

DMDC 

Defense  Manpower  Data  Center 

DoD 

Department  of  Defense 

DPS 

Deterrents  to  Participation  Scale 

DSCAC 

DMDC  Special  Cohort  and  Continuer  Files 

EDMIS 

Education  Management  Information  System 

EMF 

Enlisted  Master  File 

ESL 

English  as  a  Second  Language 

ETS 

Expiration  of  Term  of  Service 

FAST 

Functional  Academic  Skills  Training 

FFGE 

Fully  Funded  Graduate  Education 

FY 

Fiscal  Year 

GED 

General  Equivalency  Diploma 

ILIDB 

Installation  Level  Integrated  Database 

MEPCOM 

Military  Entrance  Processing  Command 

MGIB 

Montgomery  GI  Bill 

MOS 

Military  Occupational  Specialty 

NCO 

Non-commissioned  Officer 

NPV 

Net  Present  Value 

OCONUS 

Outside  the  Continental  United  States 

OLS 

Ordinary  Least  Squares 

PCS 

Permanent  Change  of  Station 

PEF 

Active  Duty  Personnel  Edit  File. 

PERSCOM 

United  States  Total  Army  Personnel  Command 

B-l 


Acronym 

Definition 

PRS 

Participation  Reasons  Scale 

SADP 

Survey  of  Active  Duty  Personnel 

SAS 

Statistical  Analysis  System 

SAT 

Scholastic  Achievement  Test 

SDT 

Self  Development  Test 

SIDPERS3 

Standard  Installation/Division  Personnel  System  3 

SOC 

Servicemembers  Opportunity  Colleges 

SOCAD 

Servieemembers  Opportunity  Colleges  Army  Degree 

SPSS 

Statistical  Package  for  the  Social  Sciences 

SRB 

Selective  Reenlistment  Bonus 

SSMP 

Sample  Survey  of  Military  Personnel 

SSN 

Social  Security  Number 

~TA 

Tuition  Assistance 

TAPDB 

Total  Army  Personnel  Database 

UIC 

Unit  Identification  Code 

VEAP 

Veterans  Educational  Assistance  Program  ~ 

VOTED 

Voluntary  Education 
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